Craig A. Croxton
Robert C. Berger
United States Air Force Academy

Numerous studies have shown no effect of journal writing on objective test scores. Most of these studies have used overall test scores as measures of journal-writing effectiveness. This paper reports the results of two studies that examined performance on specific test questions and related journal entries. We found that students who wrote journal entries on topics related to specific test questions were more likely to correctly answer those objective test questions than students who did not write on the topic.

Introduction

Instructors, teachers, professors, and educators at all levels struggle to make formal education a precursor to life-long learning. At the United States Air Force Academy (USAFA), one of the educational outcomes states, "We want to develop an attitude of intellectual curiosity in our graduates that predisposes them to lifelong learning" (italics in the original).

Many instructors have attempted to encourage critical thought, application of knowledge, and an attitude of intellectual curiosity through journal writing. The pedagogical benefit of journal writing is touted by many. The Journal Book (Fulwiler, 1987) contains many examples of the ways teachers use journals to improve student learning in such diverse disciplines as physics, chemistry, political science, and geography. Fulwiler states, "Journals are useful tools for both students and teachers. They can help students prepare for class discussion, study for examinations, understand reading assignments, and write formal papers" (p. 6).

The uses of journals in education are as varied as the instructors who use them. However, support for the educational value of these writing assignments seems almost universal. Yinger (1985) discusses many different types of journal exercises and concludes "writing is a powerful tool for learning as well as for communicating" (p. 31). Brodsky and Meagher (1987) report using journals for up to 75% of the course grade in a political science course. The journals had specific requirements and were collected frequently. They found that expressive or exploratory writing dominated the journals, and concluded that the journals improved student writing and learning by providing students different avenues to apply lessons, ask questions, and improve analytical capabilities. Grumbacher (1987) used journals in an introductory physics class and found students who were able to connect ideas in their journals (i.e., how concepts apply to their own world) were better problem solvers. Hettich (1990) also supports the use of journals as an effective avenue to enable students to relate course ideas to their experiences.

However, most of the support for journals comes from anecdotal reports and not through empirical data. McCulley (1986) has stated that the literature is replete with general speculations about the effectiveness of writing on learning, but we know little about the specifics of writing to learn. For example, we do not know what content can be taught through writing (McCulley, 1986). In fact, the impact of journal writing on learning as measured by the results of classroom testing has yielded less than encouraging results. Day (1994) divided an introductory sociology class into two random groups. One group received points based on attendance while the other group received points for keeping a journal on course material. Her study found whether a student completed the journals or merely was required to attend class did not predict essay or multiple-choice scores. Harchelroad and Rheinheimer (1993) used journals in a summer math course and found the students in the control group did better than the students who wrote journals. An interesting exception to this was the students with the lowest entering math skills who wrote journals scored as well as the non-journal writers. Jensen (1987) used journals in a year-long physics course. For the fall semester he required students in one section to keep a journal whereas students in the other section did not. For the spring semester, he reversed the requirements for the two sections. He found no differences in test scores at the end of either semester on objective, problem solving examinations. Selfe, Petersen, and Nahrgang (1986) reported the results of an experiment using journals in a ten week long college-level math course. One of the sections used journal and tests, one used quizzes, and one section used only tests as evaluations. All three sections took the same tests. Selfe et al. found no differences in objective test scores among the three sections.

Assuming we can accurately measure student learning through classroom testing, the above results seem to indicate journal writing is not improving student learning as measured by objective test scores. As Hettich (1980) notes, reading and commenting on student journals is a time consuming task for the instructor. If student learning, as measured by classroom testing, is not improved through journal writing, what is being gained by the time and effort being spent by both teacher and student?

One consistent approach to the research on the effectiveness of journal writing on objective test measures is that journal writing has been compared with overall test scores, not on correct answers for individual test items. Britton, Burgess, Martin, McLeod, and Rosen (1975) suggest when people write about new information they learn and understand the information better. If this is true, then perhaps the learning needs to be analyzed on a more basic level, such as looking at individual test questions. We wondered if the results might be different if we compared a journal entry written on a specific topic with a test item on the same topic. For example, if a physics student wrote a journal entry on friction, would the student be more likely to answer test questions related to friction correctly than a student who had not written on friction? Our study was designed to look at this level of analysis using a leadership course currently taught at USAFA. We hypothesized that writing on a given topic would lead to higher scores on examination questions on the same topic.

Study 1

Method
The Behavioral Sciences and Leadership Department at USAFA teaches a semester-long (seventeen week) junior/senior level course entitled "Leadership Concepts and Applications." This course is currently taken by approximately two thirds of cadets at USAFA. One of the course requirements is the writing of leadership application papers. These papers are reflective journals in which the students are required to produce 6-10 entries, each approximately three pages in length. The purpose of the entries is to encourage students to relate course concepts to their experiences and reflect on the lessons learned from these experiences. For example, on the topic of communication, the student might write about her commander's poor use of communication in the squadron, analyze what was wrong with the communication using a systems model of communication discussed in class, and then discuss how she would improve on the communication if she were the commander. The entries were graded pass/fail. A passing grade was given to an entry which accurately reflects course content, shows depth in reflection, and was relatively free of grammatical and spelling errors. All students were allowed one attempt to re-write a failing entry to a passing level. Students chose from 16 topics and were required to write 10 passing entries to receive 100% on their journal grade for the semester, pass nine entries for 95%, pass eight entries for 90%, and so on, down to five passing entries for 65%. Five passing entries was the minimum number needed to pass the course. The overall journal grade accounted for 20% of the grade in the course.

Participants
Participants were 170 students enrolled in the leadership course during the spring semester 1995.

Design and Procedure
Instructors were asked to keep a log of their students' entries during the semester. Each of these entries was written on a specific topic, such as communications or conflict. Some of the papers were not on a testable course topic (i.e., a paper on a leadership issue in the news) and these papers were not used in the study. We then classified test questions from two course examinations and the final examination based on the topic addressed by the question. If a question covered more than one topic or was related to a topic not included among the 16 journal topics, then that question was not used in the analysis. All test questions were multiple choice or true/false questions. Eight topics had both a journal entry and at least one related test question. The topics were 1) The Leader-Follower-Situation Model, 2) Power, 3) Communication, 4) Conflict, 5) Values, 6) Motivation, 7) Stress, and 8) Contingency Theories of Leadership. Of 105 objective test questions in all three exams, we were able to classify 48 of the test questions that related to specific journal topics.

Results
Two percentage scores were calculated for each student. For each student the test questions were divided in two groups. One group was comprised of test questions related to journal topics the student had written on, the other group was test questions related to journal topics the student had not written on. If our hypothesis is correct we would expect that students scored higher on test questions related to topics they wrote on.

A matched sample t test showed the difference between the two means (67.1% vs 63.7%) was significant (t = 3.57, df = 169, p < .001).

Discussion
Study 1 supports the idea often expressed anecdotally in the literature that journal writing enriches student understanding. Students who wrote a journal on a specific topic were more likely to get an associated test question correct than if they had not written on that topic. However, this study is not consistent with previous empirical studies cited in the introduction that found no relationship between writing journals and objective test performance. In Study 2 we collected data from a another semester of journal writing in the same course to see if our results in Study 1 were replicable.

Study 2

Method
The method for Study 2 was the same as in Study 1.

Participants
Participants were 165 students enrolled in the leadership course during the fall semester 1995.

Design and Procedure
The design and procedure were the same as in Study 1.

Results
As in Study 1, two percentage scores were calculated for each student. For each student the test questions were divided in two groups. One group was comprised of test questions related to journal topics the student had written on, the other group was test questions related to journal topics the student had not written on.

A one-tailed matched sample t test showed the difference between the two means (76.2% vs 74.0%) approached significance (t = 1.61, df = 164, p = .054).

Discussion
Study 2, although not as convincing as Study 1, does provide additional support for the hypothesis that journal writing can benefit objective test performance. The difference between the means of the test questions students had written journals on compared to the test questions they had not written journals on, approached significance.

General Discussion

Why did our study find a benefit for writing journals when previous studies did not? One explanation may be that our data analysis was more focused than those in previous studies. We only examined data specific to the journals that were actually written. However, previous studies looked at overall effects and did not limit their analysis to questions related to topics students wrote on. This difference may explain why the small advantage (our percentage differences were in the 2-3% range) writing journals provided on objective performance in our studies, may have been obscured in previous research.

Another explanation of why our study found an advantage for writing journals was the journal assignment itself. Our journals specifically require that a student reflect on their own life experiences. The instructions in the course syllabus ask the student to "describe an experience, explain how it relates to the course material, and discuss what you learned through your reflection on the experience."

One possible explanation of the results of this study may be the self-reference effect (Rogers, Kuiper, & Kirker, 1977). Rogers et al. found that when subjects evaluated word lists using structural, phonemic, semantic, or self-referent (asking if the word describes themselves) tasks, recall performance for the words in the self-referent conditions were much higher than in any of the other conditions. Rogers et al. suggested that this type of encoding results in more enduring memory because of the initial depth of processing. Students in this course were encouraged to write entries related to their personal experience and this may have resulted in deeper processing for the journal topics than for those students who only read and listened to lectures on the same topics. The deeper processing, and resulting enduring memories may have led to better performance on test questions related to journal topics.

Future work on journal writing and objective test performance should focus on whether journal writing improves performance on test questions designed to test higher levels of student learning. Bloom (1956) has developed a taxonomy of cognitive learning in which test questions can be categorized as testing knowledge-, comprehension-, application-, analysis-, synthesis-, or evaluation-level learning. The literature suggests that the benefits of journal writing may be more evident at higher levels of learning (Yinger, 1985; Hettich, 1990).

References

Bloom, B. S. (Ed.). (1956). Taxonomy Educational Objectives: Cognitive domain. New York: David McKay.

Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. (1975). The Development of Writing Abilities. London: MacMillan.

Brodsky, D., & Meagher, E. (1987). Journals and political science. In T. Fulwiler (Ed.), The Journal Book (pp. 375-396). Portsmith, NH: Boynton/Cook.

Day, S. (1994). Learning in large sociology classes: Journals and attendance. Teaching Sociology, 22, 151-165.

Fulwiler, T. (1987). The Journal Book. Portsmith, NH: Boynton/Cook.

Grumbacher, J. (1987). How writing helps physics students become better problem solvers. In T. Fulwiler (Ed.), The Journal Book (pp. 322-329). Portsmith, NH: Boynton/Cook.

Harchelroad, J. L., & Rheinheimer, D. C. (1993). Journal writing: An analysis of its effectiveness in a college- level developmental mathematics class. Research and Training in Developmental Education, 9, 55-63.

Hettich, P. (1980). The journal revisited. Teaching of Psychology, 7, 105-106.

Hettich, P. (1990). Journal writing: Old fare or nouvelle Cuisine? Teaching of Psychology, 17, 36-39.

Jensen, V. (1987). Writing in college physics. In T. Fulwiler (Ed.), The Journal Book (pp. 330-336). Portsmith, NH: Boynton/Cook.

McCulley, G. A. (1986). Research in writing across the curriculum. In A. Young & T. Fulwiler (Eds.), Writing Across the Disciplines (pp. 42-48). Upper Montclair, NJ: Boynton/Cook.

Rogers, T. B., Kuiper, N. A., & Kirker, W. S. (1977). Self-reference and the encoding of personal information. Journal of Personality and Social Psychology, 35, 677-688.

Selfe, C. L., Petersen, B. T., & Nahrgang, C. L. (1986). Journal writing in mathematics. In A. Young & T. Fulwiler (Eds.), Writing Across the Disciplines (pp. 192-207). Upper Montclair, NJ: Boynton/Cook.

Yinger, R. (1985). Journal writing as a learning tool. Volta Review, 87 (5), 21-33.

Author Contact Information:
Maj Craig A. Croxton
Lt Col Robert C. Berger
Department of Behavioral Sciences and Leadership
2354 Fairchild Drive / Suite 6L47
United States Air Force Academy, CO 80840
email: croxtonca.dfbl@usafa.af.mil
email: bergerrc.dfbl@usafa.af.mil
phone: 719-333-3860
phone: 719-333-2514