Monthly Archives: October 2015

Paper #4 Theories and Methods

Posted on October 29, 2015 | 1 comment

Introduction

In Paper #3 I proposed the Objects of Study for multimodal composition to be twofold: new media objects (websites, images, videos) and student writing. While multimodal composition usually focuses on student production of new media, the Objects of Study in this case deal specifically with analysis, how students interpret new media objects and how teachers and researchers analyze student writing on new media as data. In this paper I outline theories and methods related to a social semiotic approach to multimodal composition. I will first provide a general overview of systemic functional linguistics (SFL) theory, and then relate these methods to analyzing new media and student writing.

SFL Theory

A table showing resources for SFL

Table 1. SFL Resources (Schleppegrell)

A functional grammar is not just concerned with labeling grammatical elements according to the syntactic category they represent (nouns, verbs, adjectives, etc.) nor with identifying the role that these different elements play within a sentence (subject, object, etc.). Rather than analyzing linguistic structures in isolation or as abstract entities, a functional approach identifies the configuration of grammatical structures which is typical of or expected in different kinds of socially relevant tasks and links those linguistic choices with the social purposes and situations that “texts” (spoken or written) participate in. (Schleppegrell 45)

Functional grammar is not concerned with labeling the “formal” properties of language, instead looking at how meaning is made and the meaning options in social contexts. This functional approach to language can be very useful in education because “teachers also need a better understanding of the features of the language they aim for students to develop, and so the focus here is on the forms that language takes in academic contexts” (Schleppegrell 44). Providing a basic overview of functional grammar, language is used to create three different kinds of meaning: ideational, interpersonal, and textual. In terms of studying social context, or the register of a text, these variables are the “field (what is talked about), tenor (the relationship between speaker/hearer or writer/reader), and mode (expectations for how particular text types should be organized)” (Schleppegrell 46).

Within each of these functions is a set of tools that can be used for text analysis (See Table 1 above). In functional grammar, nouns and noun phrases are called participants, verbs are called processes (there are several kinds of verb processes: doing, saying, thinking, being, etc.), and adverbial phrases of time, place, and manner are called circumstances (see Table 2). By identifying the functional processes of language, research shows that students, including ELLs, are better able to interact with language and content on more meaningful levels than simply labeling sentences using formal grammatical rules (de Oliveira and Schleppegrell 20). Students can more readily understand the action of texts, the doers and the doings, by identifying the participants, processes, and circumstances.

Shows functional labeling of sentences in SFL

Table 2. Functional Labeling of Sentences

Multimodal Analysis

Figure 1 Semiotic Toolkit Wise

Table 3. Semiotic Toolkit (Wise)

Researchers such as O’Toole, Kress, van Leeuwen, Lemke, Baldry, Thibault, and O’Halloran have adapted Halliday’s SFL metalanguage to analyze multimodal texts (images, videos, the Web, etc.). Their research is interdisciplinary, combining work in visual literacy, social semiotics, functional grammar, and design. Table 3 above is the visual analysis toolkit I created for my students in FYC based on Kress and van Leeuwen’s multimodal social semiotic framework in Reading Images: The Grammar of Visual Design.

This visual analysis framework shares many similarities with the SFL framework for textual analysis. Both are organized around the metafunctions. Examining the ideational metafunction, students label the participants of an image as actor and goal. Instead of processes, students identify vectors, the visible or invisible oblique lines formed from participants’ gaze, body parts, and objects. Vectors are the verbs of the visual world, expressing action. Moving to the interpersonal metafunction, students identify whether participants are looking directly at the viewer in a demand gaze, or looking away in an offer. Modality is a very useful tool, which is a way of judging the authenticity of an image on a sliding scale (see fig. 1). High modality, like black and white images, conveys a truthfulness, that the image is unaltered and authentic, while low modality refers to images that look unreal, highly saturated pictures, often photoshopped, such as on the cover of fashion magazines. For a full explanation of the visual analysis toolkit, you can download this PowerPoint.

The scale of modality

Figure. 1 Modality Scale

Analyzing Student Writing on Multimodal Texts

Below I have provided a student’s recent visual analysis of a Ray Rice interview with his wife, Janay, on the Today Show addressing a domestic violence incident. I will not provide an in-depth analysis of this student text using SFL, though that would certainly be possible. Instead, I am interested in a more holistic rhetorical analysis and the effects learning a metalanguage has on student writing. First of all, I am surprised at the high level of detail and description in the analysis. The student vividly describes the participants, their mannerisms, gaze, as well as the textual features and arrangement of the screen. Secondly, the student uses the Aristotelian appeals and visual toolkit in creative ways. As I mentioned in Paper 3, students do not always uses the metalanguage in the “correct” way, for instance, saying Janay Rice is the Pathos of the image, yet students’ utilization of the metalanguage is always illuminating. The in-depth analysis is not just indicative of one student. All students that learn the metalanguage begin producing in-depth descriptions and analyses, which backs up the research that learning a metalanguage increases students’ powers of noticing (de Oliveira and Schleppegrell 34). While a rhetorical analysis is definitely necessary for students to analyze purpose, context, audience, etc., learning a metalanguage facilitates in-depth analysis of the complementarity of multimodal images:

The image used in the article “Janay Rice – Naughty or Nice” by Rachel Blitchington is of Ray Rice and his wife Janay being interviewed by the Today Show about his recent display of domestic violence. Janay is on the left side. She is wearing an orange shirt, her hair is down and slightly curled and her expression shows that she is somewhat distant from the interview, almost as if she was remembering something. Ray is on the right side. He is wearing a red shirt. He looks as if he is engaging in conversation with the interviewer. In front of both of them, a blue headline is stretched across the scene. In white lettering the headline states, “Ray Rice Speaks Out: NFL player on elevator incident and wife.” Inside the blue banner to the right is the Today Show’s logo ad slightly underneath that is the time. Underneath the blue headline there is a thinner white headline that states, “Boston marathon bombing suspect filed another request,” a completely unrelated topic. Above the blue headline there is a very thin orange banner with the words “Today Exclusive.”

Ray’s wife, Janay creates a vector through her gaze. She is not looking directly at the camera and through Logos, we can infer that she is embarrassed or even scared because of her facial expressions. She is the Pathos of this image. A domestic violence victim appeals to the public’s emotions because we sympathize with what she has gone through. Ray himself also creates a vector through his gaze. The goal of his vector is most likely the person who is interviewing him. Neither of the people shown in the image is creating a demand, they are not demanding the viewer’s attention. The reactor of this image would be the public, they are reacting to the news of domestic violence, especially because he is a NFL football player and someone who is, or was a respected member of the community. There is low modality in this image because it is in color, which shows little truthfulness. The Today Show would be showing Ethos, they are gaining credibility by having the perpetrator himself on their show.

Conclusion

At the moment, I do not see SFL or multimodal social semiotics holding much sway within English Studies. Although Kress has become very influential and authoritative within English Studies, his work within SFL has not translated to the composition community, perhaps, because composition is still operating within a Chomskyan formal approach to grammar. SFL and multimodal social semiotics have become extremely popular through much of Australia, Asia, and Europe, however. SFL has begun to make in-roads into public education in the U.S., as the literacy requirements of Common Core are demanding, and SFL offers cheap and functional solutions. The integration of SFL in the U.S. is due to a handful of researchers in genre pedagogy, such as Mary Schleppegrell. Genre Pedagogy Across the Curriculum: Theory and Application in U.S. Classrooms and Contexts is the first book to deal with a genre approach to language instruction in U.S. classrooms.

As new media becomes the dominant form of text, researchers in the U.S. will need to develop more responsive approaches to new media, not only focusing on production, but interpretation as well. As the need for interpretative frameworks is realized, SFL will be there as an already established, internationally recognized metalanguage for analyzing new media. As an added benefit, researchers in the U.S. can begin exploring functional approaches to language, breaking the stranglehold that Chomskyan, formal linguistics has had on the U.S. for half a century.

I recently had a conversation with Dr. Joanne Scheibman, a professor in the Applied Linguistics program at Old Dominion University, in which we discussed functional approaches to language. Though not a Hallidayan, Dr. Scheibman’s functional approach overlaps with Halliday in many ways. Her research, as well, differs from the “major Chomskyan paradigm.” She does not see humans as having a specific language acquisition device, “elevating a biological or built in component” to language learning. Instead, language learning is “usage based,” meaning structures arise out of social contexts and interaction. I was able to see many analogies in Dr. Scheibman’s work as a discourse linguist to work in multimodal social semiotics. Analyzing the minutiae of conversations is similar to analyzing the smallest details of multimodal texts to come to a better understanding of the social structures, which shape texts.

Works cited

De Oliveira, Luciana C., and Joshua Iddings, eds. Genre Pedagogy Across the Curriculum. Bristol: Equinox, 20

De Oliveira Luciana C., and Mary Schleppegrell. Focus on Grammar and Meaning. Oxford: Oxford UP, 2015. Print.

“Dr. Joanne Scheibman.” Online interview. 8 Oct. 2015.

Kress, Gunther R., and Theo Van Leeuwen. Reading Images: The Grammar of Visual Design. 2nd ed. London: Routledge, 2006. Print.

Schleppegrell, Mary. The Language of Schooling: A Functional Linguistics Perspective. Mahwah, NJ: Lawrence Erlbaum Associates, 2004. Print.

 

1 Comment

Posted in Uncategorized

Tagged

PAB #8 “Using a Functional Linguistics Metalanguage to Support Academic Language Development in the English Language Arts.” Jason Moore and Mary Schleppegrell

Posted on October 22, 2015 | Leave a comment

This article reports on a study using systemic functional linguistics (SFL) as a metalanguage in primary grades to help students develop a deeper understanding of characterization as well as to promote dialogic interaction within the classroom. The researchers used Martin and White’s appraisal analysis, specifically positive/negative evaluation and turned up/turned down (graduation), and the process types, “the ‘happenings’ in a text” (sensing, doing, being, saying) to help understand direct and indirect characterization, how characters “show” and “tell” their emotions and motivations (Moore and Schleppegrell 97).

Using a Metalanguage in the Classroom

Metalanguages of English Language Arts – Literary, Traditional Grammar, and Functional

When reading and discussing literature, teachers and students have a literary metalanguage (terms such as symbol, metaphor and characterization) to help make meaning of stories and discuss author’s craft. When responding to writing, teachers often use the metalanguage of traditional grammar in service of improving the “correctness” or “mechanics” of student writing. (Moore and Schleppegrell 93)

In education we most often use literary and traditional grammar metalanguages, but rarely do we employ a functional metalanguage, such as SFL, “that connects language forms to meanings in contexts of use” (Moore and Schleppegrell 93). A literary metalanguage can provide a close reading of a text, often devoid of context, while a traditional grammar metalanguage can only focus on correctness, whether a students’ writing is “right” or “wrong.” What I find so interesting about Schleppegrell’s view of the functional metalanguage is that it provides a way of uniquely interfacing with the content of a text, providing new avenues and areas of analysis. I am also interested in the ways in which classroom discussion and dialog can be fostered once students have a working knowledge of the SFL metalanguage.

Besides giving teachers and students a common language for discussing and analyzing texts, which goes beyond literary metaphors and traditional grammar, the metalanguage can also be used by researchers to analyze students’ texts. Here, student writing becomes important data for researchers. As I am researching how students use the SFL metalanguage in relation to visual analysis, I can use the SFL metalanguage to analyze their writing.

What does the SFL metalanguage have to do with multimodal visual analysis?

So, what is the connection between students in primary school learning SFL to better understand characterization and multimodal visual analysis? Both textual analysis and visual analysis use SFL as a metalanguage. For instance, the functional labels above – participant, process, polarity, etc. are used in visual analysis as well (see Table 1). My view is that students can first learn the SFL metalanguage through doing something “fun,” like analyzing commercials and websites, and then learn how the metalanguage can be applied to texts, “drawing on the SFL notion of movement back and forth along a mode continuum” (Moore and Schleppegrell 94). SFL allows for this shifting of modes, from visual to textual analysis and even to discourse analysis.

Integrating Textual and Visual Analysis in the Classroom

I have not yet been able to fully integrate visual and textual analysis as a mode continuum in the classroom, instead mainly focusing on visual analysis, although I have made it clear to students that the metalanguage they are using for visual analysis comes from functional grammar. Textual analysis can be integrated more easily in primary and secondary school settings, where teachers have more time with students, not to say that integrating SFL textual analysis is impossible. In the future, I would like to develop worksheets and create activities, which use SFL to interact with content meaning. Although most of this research has been done in primary grades, it would certainly be useful on the college level. If there was a greater push for SFL nationally, especially to prepare students for the rigorous standards of Common Core, students would become familiar with SFL at a young age, creating positive literacy development for the future.

Works Cited

Moore, Jason, and Mary Schleppegrell. “Using a Functional Linguistics Metalanguage to Support Academic Language Development in the English Language Arts.” Linguistics and Education 26 (2014): 92-105. Web.

 

Leave a comment

Posted in Uncategorized

Tagged

PAB #7 “What’s in a Click? A Social Semiotic Multimodal Analysis Framework for Website Interactivity.” Elisabetta Adami

Posted on October 22, 2015 | Leave a comment

Introduction

While there has been a fair amount of research into hypertextuality, the intertextuality or linking of digital texts through hyperlinks (see Landow), there is less research on the interactivity of digital texts, the ways in which the user interacts with the loci of “‘interactive sites/signs’, such as links, buttons and fields, which enable users to act upon the text” (Adami 3). Adami develops a social semiotic multimodal framework, adapting the metafunctions (ideational, interpersonal, and textual) to analyze the interactivity of websites. Adami proposes that “interactive sites/signs” on the web have a twofold nature; they are semiotic signs, representations, but they are also interactive and can be acted upon by users.

Syntagmatic and Paradigmatic Dimensions of Websites

Shows the syntagmatic and paradigmatic meanings in a sentence.

Figure 1 Syntagmatic and Paradigmatic Dimensions of Meaning

Syntagmatically, the interactive site/sign makes meaning in combination with other elements within a syntagm of the text displayed on the screen, and paradigmatically, it functions through selection, thus enabling the actualization of one out of a range of possible textual realizations. (Adami 6)

To describe website interactivity, Adami borrows from semiotics the concept of syntagmatic and paradigmatic meaning (see Fig. 1). Syntagmatic meaning, on the horizontal axis, is the syntax, how words combine to create sentence-level meaning. The paradigmatic dimension, the vertical axis, is concerned with the possible or available meaning options within a sentence. Adami applies syntagmatic and paradigmatic meanings to analyze the interactivity of websites. The syntagmatic level refers to the meanings created on the webpage, the organization of the page, image, text, navigational links, and the interactive sites/signs, while the paradigmatic level maps the connections linking the webpage to other sites. For instance, a search bar works paradigmatically by taking the user to other parts of the webpage and the web by typing in search terms.

Analyzing Website Interactivity with the Metafunctions

Table 1: The three metafunctions mapped onto the two dimensions of interactive sites/signs.

Table 1 The three metafunctions mapped onto the two dimensions of interactive sites/signs.

Adami creates a framework for analyzing website interactivity by mapping the metafunctions onto the syntagmatic and paradigmatic dimensions of meaning. This provides a framework for analyzing metafunctional meanings within webpages and across pages, as well as looking at the interactions between the user and the signs/sites. On the ideational level, one can analyze the semiotic meanings of signs and the actions/effects the user has upon a sign. The interpersonal function looks at construction of the identity of the site producer and user and the power relations between producer and user. The textual function analyzes the arrangement of the text (information value and salience) and the traversal of a text from one page to another. Adami adds a fourth function, the interactive value, which looks at interactivity from an aesthetic and structural level.

Discussion

This article provides a method for viewing the interactivity of websites, an important and often overlooked dimension of analysis. While I have worked with my students on describing the representational aspects of websites using metafunctional analysis, I have not yet had them explicitly analyse websites from an interactive level, though interactivity often comes up organically in analyses and discussions.

It is interesting to note the twofold nature of web icons as signs and interactive loci that perform actions. Signs used to be solely representational, but now we interact with signs on a daily basis, navigating websites, searching, communicating with others, making purchases, writing, designing . The concept of syntagmatic and paradigmatic meaning is extremely useful for analyzing representation as well as the interactive and intertextual nature of websites. Along with this article, Jay Lemke’s work in traversals is very useful. Unlike the reading paths discussed by Kress, which charts how a user creatively navigates a webpage, traversals are the ways in which users traverse their online reading experience, navigating from one page to another. More work needs to be done in the ways we traverse digital texts, as well as the affordances that interactive sites/signs provide us.

Works Cited

Adami, E. “What’s in a Click? A Social Semiotic Framework for the Multimodal Analysis of Website Interactivity.” Visual Communication 14.2 (2015): 133-53. Web.

Leave a comment

Posted in Uncategorized

Tagged

Paper # 3 Objects of Study OoS

Posted on October 15, 2015 | 2 comments

Introduction

At the moment I do not see multimodal composition having an agreed upon object of study. As Claire Lauer points out, we need “precise definitions” in multimodal composition, but we also need precise objects of study (39). The lack of precision in terminology and objects of study has to do with the fact there is no metalanguage to discuss and analyze multimodal texts in rhet/comp.

While a rhetorical approach to multimodal texts is essential to understanding new media, rhetorical analysis can only attend to new media objects (Manovich’s term) on a holistic level, the broad analysis of a text’s purpose, audience, genre, and context (14). A rhetorical analysis cannot, for instance, explain intersemiotic complementarity, how the multiple modes of a new media object combine to create new, multiplicative meanings. For this type of in-depth analysis, a functional, social semiotic metalanguage is necessary.

Secondly, the objects of study in multimodal composition have been imprecise because the focus has not been on analyzing new media objects, it has been on producing them. Using technology is not really an object of study. Additionally, the emphasis on technology leads to confusion over how to assess students’ multimodal projects.

I propose the objects of study in multimodal composition are twofold: new media objects and student writing. Student writing could be defined as traditional or new media texts, though I am focusing solely on traditional writing. To analyze these objects of study, I draw from multimodal social semiotics and genre pedagogy.

Object of Study #1: Analyzing New Media Objects

When incorporating new media in the classroom, students first have to be able to analyze new media. Multimodal social semiotics is a thriving field which utilizes systemic functional linguistics and a social semiotic framework to describe and analyze new media. By drawing on multimodal social semiotics, teachers can provide students with a framework and toolkit for analyzing new media, while also introducing a functional metalanguage that positively benefits students’ literacy and increases students’ powers of noticing (de Oliverira and Schleppegrell 16).

The most popular new media objects to be analyzed in the classroom are websites, images, and videos. In the FYC course I teach, the major unit focuses on consumerism and advertising, in which students research the advertising strategies of a brand or celebrity and write an annotated bibliography, literature review, and argumentative paper on a controversial aspect of a brand’s marketing. This unit allows for the in-depth analysis of all kinds of new media objects, including websites, print ads, commercials, and social media.

Websites are a rich source of analysis, especially with branding, because websites are the face that a person or corporation wants to present to the world. Hence, websites are the lens from which corporations or individuals present their ideology. There has been a fair amount of research on visually analyzing websites by Jay Lemke, Gunther Kress, Theo van Leeuwen, and others. In “Travels in Hypermodality,” Jay Lemke provides a visual analysis of NASA’s Goddard Space Flight Center website. Lemke uses Halliday’s metafunctions (ideational, interpersonal, and textual) to guide his analysis, but Lemke recasts the metafunctions as presentational, orientational, and organizational meaning (3). Visual analyses of websites are usually incredibly descriptive and detailed. Lemke goes into a massive amount of detail, annotating and analyzing the content of the website – hyperlinks, images, navigational features.

Kress and van Leeuwen provide a semiotic framework in Reading Images: The Grammar of Visual Design, which works well for describing websites, images, and video. Each metafunction contains tools that can be applied in a semiotic analysis. Along with the Aristotelian appeals (Ethos, Pathos, Logos), students can apply the semiotic toolkit for a deeper analysis of new media objects (see fig. 1). For instance, with the ideational metafunction, students label participants as actor and goal, look at logical and taxonomic relationships, and see how vectors create action in visual narratives. With the interpersonal metafunction, students label whether participants are using a demand gaze or offer, distinguish the modality of the image-text (how truthful the image seems), and look at the coding orientation (what group the text was made for – scientists, consumers, artists, etc.). The textual metafunction views the arrangement of a screen-text, and the tools here relate to information value – given/new, ideal/real – as well as typographic features.

Figure 1 Semiotic Toolkit Wise

Figure 1 Semiotic Toolkit (Wise)

Object of Study #2: Student Writing

The second object of study pertains to how students write about new media. I believe this skill is crucial, even more so than producing new media, especially in the composition classroom. Here I am referring to traditional writing as the object of study, specifically, how students take the theoretical framework and metalanguage outlined above and recontextualize it into their writing practices.

This new object of study is informed by genre pedagogy and discourse analysis. In a previous post, I outlined the Onion of Critical Analysis, which could be productively applied to an analysis of students’ writing on new media. The Onion consists of four levels working in a spiral, each level supporting the next – description, analysis, persuasion, and critique (Humphrey and Economou 41). At the most fundamental level, students must describe websites, images, and videos. This mostly takes the form of narrative description, summarizing an image-text, as one would summarize a scholarly article. Next, students analyze new media, using the Aristotelian appeals and the semiotic toolkit. Third, students analyze the persuasive features of texts, as well as use these texts in their argument to persuade the reader. Finally, the most sophisticated skill, students critique new media, building upon the other analytical skills.

What I have found most interesting while researching this object of study is the creative, generative ways in which students apply the metalanguage to an analysis of their new media sources. Students use the toolkit in ways I would never have thought to apply them because I am used to using these terms only in the ways discussed by Kress, Lemke, and others. Without this theoretical grounding, students are much more likely to apply the metalanguage in unique ways, making insightful, new connections. The metalanguage also helps students become detail oriented, as they scrupulously analyze the ideational, interpersonal, and textual meanings of new media.

Conclusion

By defining the objects of study of multimodal composition to be twofold – new media objects (websites, videos, images, etc.) and student writing – we can come to a precise definition of the field. The two objects of study tie directly into the two major questions: the “what” and “how” of multimodal pedagogy.

A purely rhetorical approach does not go into the depth necessary to analyze new media, so a metalanguage of new media is essential. Interestingly enough, systemic functional linguistics, the most utilized metalanguage for new media, is a rhetorical grammar, much more rhetorical than the prescriptive, rules-driven grammar so often used in composition, as Halliday makes clear in “Ideas About Language” (22). By adding just a few more terms to our rhetorical lexicon, such as modality, given/new, ideal/real, and organizing our analyses from the tri-focal perspective of the metafunctions (ideational, interpersonal, textual), which directly relate to logos, pathos, and ethos, we arrive at a precise, in-depth rhetorical metalanguage for new media (see fig. 2).

Figure 2 The Relationship Between The Aristotelian Appeals and The Metafunctions Wise

Figure 2 The Relationship Between The Aristotelian Appeals and Metafunctions (Wise)

Researchers have been using a social semiotic metalanguage for over two decades, but it has yet to be well-known or utilized within rhet/comp, which is still mainly technology-driven and product-focused. It is essential we give students a way of critically analyzing new media, so, as Douglas Rushkoff says, students become the programmers, not the programmed (“Program or be Programmed: Ten Commands for the Digital Age”).

Works Cited

De Oliveira Luciana, and Mary Schleppegrell. Focus on Grammar and Meaning. Oxford.: Oxford UP, 2015. Print.

Humphrey, Sally L., and Dorothy Economou. “Peeling the Onion – A Textual Model of Critical Analysis.” Journal of English for Academic Purposes 17 (2015): 37-50. Web.

Lauer, Claire. “Contending with Terms: ‘Multmodal’ and ‘Multimedia’ in the Academic and Public Spheres.” Multimodal Composition: A Critical Sourcebook. Ed. Claire Lutkewitte. Boston: Beford/St. Martin’s, 2014. 22-41. Print.

Lemke, J. L. “Travels in Hypermodality.” Visual Communication 1.3 (2002): 299-325. Web.

Kress, Gunther R., and Theo Van Leeuwen. Reading Images: The Grammar of Visual Design. London: Routledge, 1996. Print.

Manovich, Lev. The Language of New Media. Cambridge, MA: MIT, 2002. Print.

 

2 Comments

Posted in Uncategorized

Tagged ,

PAB #6 “Systemic Functional-Multimodal Discourse Analysis (SF-MDA): Constructing Ideational Meaning Using Language and Visual Imagery.” Kay L. O’Halloran.

Posted on October 8, 2015 | Leave a comment

Introduction

In other posts I have outlined a general theory of systemic functional multimodal discourse analysis. Based in Halliday’s systemic functional linguistics (SFL) and adapted to analyze alternative semiotic systems such as image and sound by O’Toole, Kress, van Leeuwen, and others, SF-MDA applies the metafunctions, the ideational (experiential and logical), interpersonal, and textual to an analysis of images and screen-based texts.

I have to admit, I am still digesting O’Halloran’s article. She goes into an extremely complex level of detail, adapting the nuances of SFL to an analysis of imagery. Specifically, O’Halloran’s work deals with intersemiosis, the meaning created by the combination of multiple semiotic modes, especially the meaning that arises through the juxtaposition of image and text. This is an important field of research, as screen-texts almost always combine imagery with typography, and developing theoretical frameworks for understanding this new interpretative space is essential to multimodal research. I will explore some of these frameworks as well as discuss O’Halloran’s suggestion that multimodal texts can be effectively analyzed by using software such as Photoshop.

Analyzing Text and Images

While there has been a lot of research into using SFL to analyze an image or text, multimedia requires we develop Cross-Functional Systems, frameworks that can interpret, for instance, meaning created from the combination of image and text.

A basic theory of SFL is that language is divided into two stratums: content and expression. The content of language consists of lexicogrammar, the combination of words, word groups, clauses, clause-complexes, and mechanics, while the expression stratum consists of letters and phonology and is the way in which language is expressed.

O’Halloran adapts the content and expression stratums to imagery, especially the intersemiotic complementarity, or combination of multiple semiotic modes (see table 1).

Systemic Functional Framework for Visual Images

Table 1 Systemic Functional (SF) Framework for Visual Images

Here the content stratum is broken into discourse semantics and grammar. Like the discourse level of language, which deals with large bodies of text, the discourse semantic level in this framework looks at visual images as a whole. At the more detailed level of grammar, the framework examines the scenes, episodes, figures, and members of an image. The expression plane deals with graphics and arrangement, which convey the content to the viewer. The expression stratum requires a Cross-Functional framework because this level combines several semiotic modes to create cohesive meaning – graphics, color, image, text, etc.

A Framework for Analyzing Advertisements

Generic Structure of Advertisement

Table 2 Generic Structure of Advertisement

I have found table 2 useful for the analysis of advertisements. The table breaks advertisements into their visual and linguistic components. The visual component consists of the Lead, the Locus of Attention (LoA), the most salient figure in the advertisement, and the Display, the way in which the figure is displayed. Secondly, the table analyzes the linguistic components of the ad. Advertisements usually have an announcement followed by primary and secondary text. An ad may also have an enhancer, text which enhances the primary announcement as well as call-and-visit information, which displays website, contact information, etc.

Analyzing Multimodal Texts with Digital Technology

O’Halloran suggests using digital technology such as Photoshop to analyze multimodal texts. Teachers, researchers, and students could import an advertisement into Photoshop and annotate the text as well as experiment with color saturation, light, tint, and other features. O’Halloran has since developed multimodal analysis software, but I have not yet evaluated it. Other multimodal annotation software includes ELAN.

I find both aspects of O’Halloran’s article, the need for frameworks to interpret intersemiosis as well as the use of digital technology to interpret and annotate multimodal texts, important for my work. I would most likely test out multimodal annotation software before using it in a class, but the use of multimodal annotation software brings up several questions, the most important being how to address the time and resources required to use it in the classroom. What is the educational value of using digital technology to analyze multimodal texts? At this time, I am unsure of the answer to these questions. However, using annotation software in the classroom definitely reinforces a design-based pedagogy and gives students experience with digital tools.

Works Cited

O’Halloran, Kay L. “Systemic Functional-Multimodal Discourse Analysis (SF-MDA): Constructing Ideational Meaning Using Language and Visual Imagery.” Visual Communication 7 (2008): 443-75. Web.

Leave a comment

Posted in Uncategorized

Tagged

PAB #5 “Gains and Losses: New Forms of Texts, Knowledge, and Learning.” Gunther Kress.

Posted on October 8, 2015 | Leave a comment

Framing

Kress begins the piece by stating two “central assumptions”:

Central assumptions of multimodal approaches to representation and communication are (a) that communication is always and inevitably multimodal; and (b) that each of the modes available for representation in a culture provides specific potential and limitations for communication. (5)

Palmeri speaks to this first point in Remixing Composition: A History of Multimodal Writing Pedagogy when he argues in Part One that “Composition Has Always Already Been Multimodal.” Defining multimodality as something new cheapens its rich history in writing and composition. Indeed, just holding a conversation with someone involves multiple modes of expression – speech, image, sound, and gesture. The second central assumption has to do with the affordances and aptness of media, affordances being the particular strengths and weaknesses of a medium and aptness being the appropriateness of design choices given the social situation and resources available.

In particular, it seems evident to many commentators that writing is giving way, is being displaced by image in many instances of communication where previously it had sway … This realization calls forth a variety of responses, mostly negative, ranging from outright despair, anger, and nostalgia to some still utopian voices on the other end of the spectrum. (5)

The revolution of media, the shift from print to image, has caused alarm in composition on both sides of the aisle. Some traditionalists lament the rise of electronic communication and the concurrent supposed demise of student literacy. Students’ writing has become marred by “textspeak” – slang, abbreviations, emoticons, lack of punctuation. On the other side of the aisle, the “utopian voices” wholeheartedly embrace technology, some even welcoming the demise of print.

The Logic of the Page vs. The Logic of the Screen

A comparison of IoE prospectus with IoE homepage

Fig. 1 IoE Prospectus and IoE Homepage

Kress illustrates this revolution in media by comparing a London Institute of Education promotional brochure from 1992 with a screen-image of their website from 2005. The prospectus has a fixed reading path. The document is read from left to right and ordered hierarchically in terms of the structure of the institution. The prospectus assumes the life-world of the prospective student, that “the structure of the institution and of its knowledge [are] identical with the needs of the life-worlds of the individuals who might come to it as its students” (Kress 9). The prospectus gives the impression students must conform to the rules and organization of the institution.

The semiotic landscape in 2005 is drastically different. Instead of being read from left to right, the website contains multiple entry points, ways to navigate the page. “Reading” a website becomes an act of creative design. Images are the dominant mode of text on the website, while print text serves a secondary role. The designers of the webpage make no assumptions about the life-worlds of perspective students, as the audience of the website cannot be completely determined. Also, there is much more emphasis on students and student life as opposed to the organizational structure of the institution.

From Critique to Design

It [Critique] challenges the existing configurations of power and expects that in exposing inequities more equitable social arrangements could be developed. In terms of representation that would amount – at that time when the focus was clearly linguistic – to lessening the effects of power and its realization in linguistic form … Now, in the early part of the 21st century, there is no need for bringing the social into crisis: it decidedly is. (Kress 17)

This fourth section of “Gains and Losses” has had a profound impact on my thinking. Kress charts the shift from “competency” or “mastery” of knowledge in the 1950s to the new era of critique starting in the 1960s and 70s. Kress argues that this form of social critique, meant to bring our social institutions into crisis, is now making matters worse, as all of our social institutions are already in a state of crisis. Now pointing out unequal power relations in language is the dominant form of cultural capital, yet I do not think merely pointing out disparities in language goes far enough in the indictment of unequal power relations. Indicting linguistic inequalities almost always leaves unequal power structures in tact.

Reflection

Kress’s “Gains and Losses” has provided me with a theoretical framework for multimodal pedagogy. The essay also provides a vivid description of contemporary society with implications for education, as we move from a print-based to screen-based culture, and all the cultural ramifications this shift entails.

In the classroom, I have taken from this essay to teach students the differences between the logic of the page and the logic of the screen. Specifically, terms such as reading paths, entry points, and life-worlds, as well as navigating websites as an act of design, has framed classroom discussions and analyses of screen-based texts. Although some may worry that what Kress advocates will distract educators from the traditional goals of composition, I continue to keep in mind what Kress says, “The elites will continue to use writing as their preferred mode, and hence, the page in its traditional form” (18). Teaching writing and critical thinking is still my main concern, yet an understanding of our contemporary rhetorical situation helps create a relevant and responsive pedagogy.

Works Cited

Kress, Gunther. “Gains and Losses: New Forms of Texts, Knowledge, and Learning.” Computers and Composition 22.1 (2005): 5-22. Web.

Leave a comment

Posted in Uncategorized

Tagged

Paper #2 Major Questions: The “What” and “How” of Multimodal Pedagogy

Posted on October 1, 2015 | 4 comments

I am now turning my attention from genre pedagogy to multimodal pedagogy. My previous postings are not for naught, however. My conception of multimodal pedagogy allows for the navigation between genre pedagogy, multimodal analysis, and critical discourse analysis using systemic functional linguistics (SFL). Specifically, borrowing from my previous PAB on The Onion of Critical Analysis, I will study how students use the SFL metalanguage to describe, analyze, and critique multimodal texts. For now, I will be looking at two major questions: the “what” and “how” of multimodal pedagogy.

The “What” of Multimodal Pedagogy

The two questions – the “what” and “how” of multimodality – were first raised in the New London Group’s influential work “A Pedagogy of Multiliteracies: Designing Social Futures,” published in 1996. The NLG’s document was groundbreaking in the field of composition, framing all discussions on multiliteracies, multimodality, multimodal composition, new media, digital rhetoric, and any other fancy terms you want to call it, that followed.

First, we want to extend the idea and scope of literacy pedagogy to account for the context of our culturally and linguistically diverse and increasingly globalized societies, for the multifarious cultures that interrelate and the plurality of texts that circulate. Second, we argue that literacy pedagogy now must account for the burgeoning variety of text forms associated with information and multimedia technologies. (NLG 60)

To the first point, the NLG describe the shift in our society from the Fordism of production lines, ” a division of labor into minute, deskilled components,” to a Postfordism, global society with a horizontal power structure and “‘multiskilled,’ well-rounded workers who are flexible enough to be able to do complex integrated work,” where the workplace is centered around teamwork, and managerial discourses tend to infiltrate all aspects of social life, education, community, and the workplace (66). To their second point, the shift to a global, screen-based society requires educators to transform their theoretical frameworks and methods, embracing a diversity of literacies, the languages of diverse cultures, as well as the literacies and semiotic systems of new mediums, such as television, film, music, and the Internet, literacies that go beyond the “dominance” of standard print.

The Metalanguage of Design

“In addressing the question of literacy pedagogy, we propose a metalanguage of multiliteracies based on the concept of design” (73).

“In other words, [teachers and students] … need a metalanguage – a language for talking about language, images, texts, and meaning-making interactions” (77).

Gunther Kress and Norman Fairclough’s influence can be felt throughout the NLG’S statement. Throughout Kress’s work, he has urged for a metalanguage to describe various semiotic systems, a flexible toolkit that does not put an undue burden on teachers and students (NLG 77). Also, from Kress comes the idea of design. Students are no longer merely passive receivers of information; instead, students use the semiotic resources available to them to design and redesign texts, a design-based pedagogy taking into account all the sociocultural, historical factors that go into designing. Fairclough contributes the concept of discourse and plurality of discourses in a contemporary society as well as the ways in which power is regulated through orders of discourse.

Although the NLG has become extremely influential within rhetoric and composition, I do not see the NLG’s recommendations for a metalanguage and design-based pedagogy, the “what” of multiliteracies, being taken seriously into consideration. This is probably due to the extreme paradigm shift required for these recommendations to be implemented. Instead, multimodal pedagogy has split into two camps: multimodal composition, a product-driven, technology-focused curriculum, and multimodal social semiotics, multimodality in its original sense, utilizing SFL as a metalanguage and social semiotic framework to analyze multimodal texts, the second strand being quite rare in education.

Multimodal Composition

Two surveys show the “what” of multimodal composition boiled down to the integration of technology in the classroom and the assigning of projects in lieu of traditional writing assignments. In their survey, Anderson et al. investigate multimodality as technology use in college classrooms, which software were most prevalent, access to technology, instructional approaches to integrating technology, and the types of multimodal projects teachers assigned their students. Lutkewitte’s survey of graduate teaching assistants in first-year composition similarly focuses on technology usage and the incorporation of multimodal projects in the classroom. This is not to say that technology integration and projects are not essential. Giving students access to technology and software helps them become upwardly mobile members of the creative class, but educators need to frame the production of multimedia from a design-based perspective.

Indeed, studies such as these show there is much confusion over what multimodality even means. Claire Lauer brings up the confusion over terminology and comes to the conclusion that the term “multimodality” is used in academic settings to refer to a number of different things, mostly the use of technology in the classroom, and that “multimedia” is instead used more frequently in popular contexts outside academia (26).

Multimodal composition, as remediation, is often utilized to give basic writers non-traditional project options that connect with their personal lives, while not being overly demanding from a traditional literacy perspective. The remedial view of multimodal composition definitely differs from the NLG’s original contention that students need explicit instruction in a variety of literacies to become empowered members of privileged discourse communities. “The NCTE Position Statement on Multimodal Literacies” echoes this idea of remediation, the idea that students from impoverished backgrounds without access to literacy benefit from the integration of technology and projects. Yet, this quick fix does not address the greater need for critical literacies, especially among students from impoverished backgrounds.

Others researchers in new media and digital rhetoric have applied rhetorical and critical lenses to multimodality, using ethos, pathos, and logos, as well as feminism, postcolonialism, and cultural studies. Although these lenses are crucial, they do not provide a method for systematically analyzing various semiotics systems.

The “How” of Multimodal Pedagogy

Luckily, there has already been several decades of research into multimodality within SFL and social semiotics. The main question here is not what we do with technology, but how we analyze it. I have always found the reasoning behind the integration of multimodal projects in the classroom somewhat counterintuitive, as though using technologies helps students understand technologies’ sociocultural ramifications. For instance, it would seem absurd to require students to write a book in order to understand literature. As screen-based texts have become the dominant form of writing and communication, there is a greater need for teachers and students to be able to describe, analyze, and critique multimodal texts.

Multimodality is based in the work of Michael Halliday. Halliday made an important distinction, shifting the conceptualization of human experience from knowledge to meaning (1). This semantic shift from knowledge to meaning enables researchers to study meaning-making in various semiotic systems. O’Toole was the first researcher to apply Halliday’s functional language to an analysis of art in The Language of Displayed Art. In the semiotic toolkit O’Toole created, meaning is viewed from three perspectives (the representational, modal, and compositional) and four ranks of depth (see fig. 1). Next, Kress and van Leeuwen adapted this metalanguage for screen-based texts. Since then, researchers have been developing specialized metalanguages for many fields, including children’s books, websites, film, television, commercials.

O'Toole Metafunctions

Figure 1, O’Toole’s Semiotic Toolkit

 

One of the biggest questions to come out of this research is intersemiosis, how multiple modes, such as the juxtaposition of text and video, combine to make meaning. Jay Lemke has even researched, what I guess you would call trans-semiosis, how the meaning-making systems of large, transmedia franchises such as Harry Potter span multiple platforms – books, movies, toys, videogames, electronic spaces, etc (581).

A social semiotic approach to multimodality is useful for several reasons. It gives teachers and students a metalanguage and functional toolkit for analyzing multimodal texts. This toolkit is recognized by researchers in fields from all around the world, is not overly burdensome for teachers and students to learn, especially if the metalanguage is introduced in primary grades, and applies to many literacies – print texts, the Internet, film, and sound. The research I have done using a metalanguage and multimodal toolkit has led to some surprising results. Students that learn the metalanguage raise metawareness, pay more attention to detail, become better critical analysts, and boost confidence. There will certainly continue to be ongoing debates as to what multimodality, new media, and digital rhetoric is, but in the meantime I am interested in finding answers to the questions of how. How do we integrate a metalanguage into education that spans mediums and promotes literacy and critical analysis?

Works Cited

Anderson, Daniel, et al. “Integrating Multimodality into Composition Curricula: Survey Methodology and Results from a CCCC Research Grant.” Composition Studies 34.2 (2006): 59-83. Web.

Halliday, M. A. K., and Christian M. I. M. Matthiessen. Construing Experience through Meaning: A Language-based Approach to Cognition. London: Continuum, 1999. Print.

Lemke, Jay. “Transmedia Traversals: Marketing Media and Identity.”

Lauer, Claire. “Contending with Terms: ‘Multmodal’ and ‘Multimedia’ in the Academic and Public Spheres.” Multimodal Composition: A Critical Sourcebook. Ed. Claire Lutkewitte. Boston: Beford/St. Martin’s, 2014. 22-41. Print.

Lutkewitte, Claire. Multimodality Is…: A Survey Investigating How Graduate TeachingAssistants and Instructors Teach Multimodal Assignments in First-YearComposition Courses. Print.

NCTE. “NCTE Position Statement on Multimodal Literacies.” Multimodal Composition: A Critical Sourcebook. Ed. Claire Lutkewitte. Boston/New York: Bedford/St. Martin’s, 2014. 17-21. Print.

New London Group. “A Pedagogy of Multiliteracies: Designing Social Futures.” Harvard Educational Review 66.1 (1996): 60-93. Web.

O’Toole, Michael. The Language of Displayed Art. Rutherford: Fairleigh Dickinson UP, 1994. Print.

 

 

 

 

 

4 Comments

Posted in Uncategorized

Tagged ,