Multimodality Concept Maps and Video Documentary Reconstructions: New Uses for Adaptive Multimedia in Learning   Roger D. Ray Professor of Psychology Rollins College Winter Park, Fl.

Growing and compelling constructionist learning literature has made "active learning" a touchstone technique for many of today's educators. Likewise, concept mapping has become a widely respected assessment strategy for evaluating how students articulate the meanings they construct from such personal experiences. Unfortunately, the extant literature on concept mapping remains focused on the technique's origins as a language-based tool. For example, Novak and Gowin (1994) describe only the use of words, phrases, and qualifiers in pursuit of understanding a student's linguistic associations within a topical domain. Nevertheless, it is possible--even desirable--to articulate a broader view of concept mapping as a multi-modality knowledge representation activity.

THERE IS MORE TO CONCEPTS THAN WORDS

Closer study of knowledge representation and communication processes reveals sorely neglected possibilities for expanding the concept mapping tool set beyond the singular use of linguistic representation. One outstanding example is evident in Tufte's (1983, 1990) seminal illustrations of the power of graphic representation for conveying complex concepts and events, such as temporal and geographical landmarks attending troop losses during Napoleon's devastating Russian campaign. Mathematicians, engineers, and even musicians would quickly add symbolic (e.g., mathematical) representation schemes to the arsenal of conceptual representation and communication. And I cannot imagine an architect trying to convey the nuances of a new skyscraper concept to a civilian client without the use of physically or computer-constructed simulations or models. Unlike two- dimensional blueprints or elevational drawings, the construction of a simulated model allows direct contact with a building's three-dimensional space and proportionality, not to mention allowing a direct representation of the way the building is likely to impact the visual landscape surrounding it. Imagine the limitations if such a conversation between architect and client were constrained to words alone!

Anyone dealing with complex events and objects will quickly appreciate the ability to communicate their thoughts and knowledge by using a variety of expression modalities. At a minimum, words, graphics (including line drawings, photographs, paintings, movies, and even geographically arranged information such as tables), symbols, and constructed models will find their own power for communication and illustration.

KNOWLEDGE DIMENSIONS BEYOND REPRESENTING AND ASSOCIATING

Just as there are other modes of expression beyond language, there are also other dimensions to knowledge that have been ignored by those who stress concept mapping-dimensions that play a critical role in defining the broader scope of knowledge and reasoning skills. In the section above, I introduced the idea of alternative expression modalities and stressed the act of event and object representation through a given mode of expression. But the concept mapping literature stresses, albeit within the linguistic mode, the act of association. That is, during concept mapping exercises, students are asked to associate words with other words. For example, when one hears the words "John F. Kennedy," it is almost impossible for those of a given generation not to think also of "United States president" and "assassination." These are ingrained associations between words or phrases.

Likewise, how many problems can you think of which require you to associate between graphic representations? If given a painting by Monet and a multiple choice of four other paintings by various painters, only one of which is another Monet, could you identify the correct one? Any art historian or museum curator almost certainly could. But how often do we strive to teach our students to rely upon such non-linguistic association skills? And how often do we use nonlinguistic associations for evaluational concept mapping? If inspection of the research literature is any indication, not very often. But this only emphasizes just how much in today's educational processes we stress language to the exclusion of other modes of knowing, and thus other modes for expressing our knowledge.

Translation between modalities is a skill, which can be characterized as making associations between alternative modes of expression, as opposed to making associations only within a given mode. For example, musicians not only write music, but also perform it from the written score (a type of modeling). Again, mathematics is exemplary when students are taught to translate problems presented through words into appropriate formulas. These formulas then lead, through processes of within-domain associations, to concrete solutions to the verbal problem. It is also relevant to reflect at this juncture on how often math students are encouraged to "draw a picture" of a word problem as an intermediate translation to help generate the correct mathematical formula.

But perhaps the most powerful knowledge dimension comes when one both translates between and associates within modalities simultaneously, thereby constructing what I refer to as aggregated concept clusters. An example of such aggregation was just accomplished-I hope!-in the above paragraphs when words and mathematical formulations were used in concert to generate the concept of across modality translation as a form of knowledge communication. And that discussion presumably has much more power for the audience who also will see it modeled through a software system that embodies its principles than for those who are restricted to only reading it. The demonstration models the aggregation of various domains, not just the words. As such, the software represents my own aggregated concept structure and it includes all domains-verbal, symbolic, graphic, and constructed models-in concert at the same time.

But how many of us in the teaching profession set out to include the specific use of all such concept expression tools when we aspire to teach knowledge construction skills? And why not? I have already suggested that one limitation has been the lack of a conceptual taxonomy of component skills. But there are other limitations as well, including the lack of physical expression tools and the problem of defining appropriate problem spaces in which to develop a student's skills.

A MULTIMEDIA PROBLEM:
THE COMPLEXITY OF KNOWLEDGE EXPRESSION TOOLS

While some teachers simply miss seeing the value in teaching multiple modality thinking and expression skills, others are convinced of its relevance. Nevertheless, it still is not an easy task to assume. Besides lacking a taxonomy for guiding our goals, a second impediment has been our inability to integrate physically such an array of expression modalities. For certain, some teachers involve students in class projects where many modalities of expression and construction can come together in project constructions, as happens in engineering fairs for example. But rarely do we emphasize multiple modality expression as a matter of everyday study skill development. The difficulty of managing such a multi-modal learning process is not trivial. Tools for thinking and communicating through integrated multiple modalities of expression have not been very user-friendly until very recent times. But as multimedia technologies advance, those advances offer new opportunities and challenges for teaching in significantly new ways.

Then, computer-based word processing arrived. As noted above, cut and paste is more than an electronic metaphor-it is how we used to actually do it. And given today's word processing technology, many more teachers now expect multiple drafts of essays and term papers. These teachers now rely on rewriting as a means for teaching critical editorial skills-skills requiring one to reflect upon writing structure and choice of wording. This brings the teaching of writing to a whole new level; and it should be stressed again that this revolution has developed as a result of our incorporation of computing technology into our teaching. Many pundits who lament the stress on technology in teaching seem to have missed the significance of this non-trivial application of computers in our everyday assignments.

A growing number of technologically sophisticated teaching pioneers glimpse intuitively many of the reasons for such technology applications. Like those before them who saw the multiple editing and rewriting advantages inherent in electronic word processing over hand-written or even typed, single-version theses, these multimedia-oriented teachers see practical, if not theoretical, advantages of teaching students to express themselves through a multitude of modalities.

The use of multimedia authoring is constrained by its seeming complexity. Even teachers who get intuitively excited and motivated to use multimedia tools in their teaching are all too often discouraged by extant authoring technologies. These limitations, of course, lie with the software technologies, not the hardware technologies. Thus, many teachers, and to a lesser extent their students, are all too often intimidated by the inherent complexity of the authoring task. This is the dilemma I first faced when, despite my understanding of why I wanted to use student- oriented multimedia, I had to figure out how I could use it. That, and a number of other contributing factors, led me to develop both a new software system, with relatively simple authoring attributes, and a comprehensive strategy for incorporating its use into my teaching.

A SECOND MULTIMEDIA PROBLEM:
COMPLEXITY OF AGGREGATED CONCEPTS

I have noted that a significant problem impeding multimedia-teaching efforts is the lack of sufficiently simple authoring environments that won't detract unduly from the primary production and expression task. But another problem of almost equal importance exists as well. That is the problem of helping students find a sufficiently defined "problem space" to explore without their losing sight of the intellectual growth that comes from depth as well as breadth of exploration. Let me cite an analogy to express this problem.

So how do we get a student to better appreciate the overall picture of where we would like them to go in their next few months of intellectual growth and pursuit? And how does this relate to multimedia authoring? Despite the case for multimedia authoring by students, when students are first exposed to hyperlinked libraries of information, they are quickly and easily overwhelmed. In fact, one of the most commonly cited problems associated with the use of hypermedia libraries is that of getting lost in the web of information. Hypermedia libraries lack a coherent integrative mapping of their purpose and content, much less their inherent integration of content elements.

SEEKING SOLUTIONS TO
MULTIMEDIA TEACHING PROBLEMS

Over the past 10 years or so, my laboratory has been addressing both problems described above (i.e., lack of accessible multimedia authoring tools and defining clearly stated and meaningful problem spaces for creative expression). We started first on authoring tools by building non-scripting templates for authoring multimedia presentations in Apple computer's Hypercard. That work evolved and we now have both cross-platform and Internet versions, which incorporate highly sophisticated tool sets. The software system that grew from these efforts is called MediaMatrix.

Beginning about this same time, I also started exploring the other multimedia problem by using documentary videos as a means for motivating some independent-study students to explore a relatively finite, yet sufficiently open, "problem space." I asked students to conduct what I call "production autopsies" of NOVA or National Geographic productions.

In my "film autopsy" assignments, students demonstrate their individual or collective connections among ideas presented within a given documentary by reconstructing much of the substantial research that was required on the part of the authors and producers to bring their documentary to fulfillment. Students then archive their research products and hyperlink them to appropriate components of the video documentary. I refer to this as building a text- and-graphic "back-end database" to the digital video "front-end" presentation.

Importantly, the MediaMatrix software system allows one to connect a multitude of related objects, including full textual essays, brief pop-up footnotes, still pictures, relevant video comments, other conceptually related full-screen video topics, tabular summaries of relevant data, and even full-scale laboratory simulations that illustrate the concepts to which they are attached. This is most easily demonstrated by navigating to a textual topic, say a biographical essay on Thorndike, and shifting the software into authoring mode. Making any selected text bold, to indicate that it is a navigable hot link, also allows us to connect an entire matrix of associated topics, comments, videos, tables, or simulations.

THE RELEVANCE OF FOUNDATIONS
TO AGGREGATED CONSTRUCTIONS:THE ROLE OF TUTORIALS

It is unfortunate, in my opinion, that those who stress contructionistic learning all too often ignore the relevance of foundations. As with physical buildings, the foundation of any construction determines the overall integrity and strength of the more obvious elements. As this metaphor relates to knowledge systems, foundations are typically defined by the representation repertory of the student--the technical vocabulary and even the jargon used within a discipline of study. All too often, if a student lacks a sufficiently developed sense of fundamentals, the discipline itself becomes meaningless as it expands in associative scope and aggregative structure. But how are foundations best laid?

As such, the MediaMatrix software system was also designed to be an artificially intelligent, adaptive instructional tutoring system. If we switch modes of use and initiate the "Tutor" mode, we find ourselves in a tutorial system that adapts level of difficulty and even content of questions to meet the differing needs of each individual viewer. Based on an imbedded automated knowledge generation engine, MediaMatrix not only tracks the user's path of use, but also develops a mirror image of his/her developing knowledge network of associations as measured by the tutorial questions. This is made possible by the fact that all questions are coded for their incorporation of at least two concept terms, images, or actions. As a student develops an understanding of these associational relations, answers to questions reflect which associations are being constructed . This mirror image of the student is then used to select the type of presentation, form, and difficulty of the tutorial question to be asked, and even which content the question should stress. This is a process called adaptive instruction.

It is beyond my present purpose to detail much more about the adaptive instructional concept itself. But those interested in knowing more about how it works, why it is relevant to tutorial processes, and which pedagogical principles drove its development might want to read other articles on the topic (e.g., Ray, 1994; 1995a; 1995b; 1995c; Ray, Gogoberidze, and Begiashvili, 1995). Once I saw the fundamental merit of the multimedia approach, I immediately saw the need for building simpler, yet more powerful, authoring environments.

THE ROLE OF MULTIMEDIA AND TUTORIAL AUTHORING

I especially wanted to allow students to focus more on the content and connectedness of their production efforts, not the technology per se, which I feel is certainly important, but should remain secondary. That is, I want to use technology, not teach it. I sometimes, even somewhat sarcastically, refer to the teaching of computer technologies as being equivalent to the offering of a curriculum on the typewriter 50 years ago. Nevertheless, I do appreciate the rather significant differences between the computer and the typewriter. And this isn't to overly demean typing skills. I have always appreciated being allowed to take a course that developed my own typing skills while I was still in secondary school. I just would prefer that others teach computer technologies while I get on with teaching my own primary discipline-psychology.

I believe the ideal system should allow one to author either simple multimedia productions or complex tutorials, as the applicational ends dictate- I also anticipated that I would want to teach advanced students how to write tutorial services for slower students. This allows one group of students to reflectively construct learning opportunities for other students more in need of building conceptual foundations. Building tutorials also encourages student authors to reflect upon their own productions in highly constructive and critical ways, such as deciding which ideas are worthy of the development of questions. And if my own adaptive tutorials emphasized learning verbal and multimodality associations, then the student tutorial author would also be retrospectively "concept mapping" his/her own production.

The authoring tool we eventually created remains in constant refinement and evolution in -my laboratory. As demonstrated, MediaMatrix allows students to include textual essays as well as original video documentary material as a fully integrated system. Superimposition of navigable hot spots within that text or video illustrates a student's conceptual linkages to illustrations, expansions, or otherwise associated materials. Given that students may branch to as many choices, incorporating as many varieties of media, from each "source object" as they desire, the system allows them to construct an aggregated matrix of their multi-modality associations.

Work in progress will eventually allow teachers quick access to a summary of these various linkage matrices, thereby offering a self-reporting summary of the students' mapping of connections among various types of content and media as well as specific topics. Tutorial questions based on cross-modality associations and even video element identifications are also possible. Of course, finding and developing the electronic assets for this expression, whether via traditional library and archive searches or the Internet, are integral components in the preparation of the final composition.

CONCLUSION

I have suggested that constructionistic learning can benefit greatly from modem advances in digital technologies. I have also suggested what I believe are sound pedagogical reasons for incorporating these technologies. The literature on concept mapping is highly restrictive in its exclusive emphasis on language as the means by which we express or evaluate knowledge. Nevertheless, concept mapping points the way to a much more elaborate and sophisticated approach to knowledge expression and evaluation if we add graphic, symbolic, and simulated modeling to our linguistic arsenal of production. Likewise, we need to understand the context in which concept mapping's emphasis on associational linkages is defined. If we view knowledge as incorporating representation, across-modality translation, and multimodality aggregation, as well as the traditional emphasis on within-modality association, then we begin to appreciate the inherent power of hypermedia constructions.

But constructing hypermedia resource libraries is a daunting task if one does not have a clear organization mnemonic to keep it confined and organized. I have suggested that one of the most motivating and powerful mnemonic tools one can use is that of documentary videos. Students who reconstruct some of the archival and constructional activities behind the director's and editor's voice find unique opportunities for building new associations, modality translations, and even aggregations that give the documentary depth and breadth it never had as a singular production. In doing so, students quickly learn meta-knowledge construction skills as direct goals of the enterprise. Technology, both hardware and software, is rapidly approaching the point where such pedagogical processes should be the rule, not the exception. But that presents quite another challenge-the challenge of changing teaching strategies and behaviors. Time will only tell whether or not we can eventually meet the challenges inherent in this application of our technological tools. We definitely don't have to wait for compelling pedagogical reasons or user-friendly tools any longer.

References

Novak, J. D. and Gowin, D. B. (1994). Learning How to Learn. New York: Cambridge University Press.

Ray, R. D. (1994). Using virtual reality to deliver laboratory experiences in undergraduate education. Proceedings of the Orlando Multimedia '94 Conference. Warrenton, Virginia: Society for Applied Learning Technology.

Ray, R. D., Gogoberidze, T. & Begiashvili, V. (1995). Adaptive computerized instruction. Journal of Instruction Delivery Systems. Summer, 28-31.

Ray, R. D. (1995b). MediaMatrix: An authoring system for adaptive hypermedia teaching-learning resource libraries. Journal of Computing in Higher Education. 7 (1) 44-68.