Computers and Testing

Tom Rocklin
Professor of Educational Psychology & Director, Center for Teaching
University of Iowa

My students don't like taking tests, and I don't like giving them, but in many of my classes, I find tests are an important part of my assessment strategy. So, I give tests. The reasons I don't like giving tests have a lot to do with the fact that developing, administering, scoring, recording the results of tests, and providing feedback to students involves a lot of repetitive and mundane work. The good news is that computers excel at the repetitive and mundane. In what follows, I am going to take a look at some (and really only a few) of the ways that computers can help us in developing, administering, scoring, recording the results of tests and providing feedback to students. Permit me one warning before I continue: Whenever I think about issues at the intersection of technology, teaching, and learning, I am struck by how much the instructional context under consideration conditions the conclusions I reach. Some of the ways computers can help us with testing will be easy to implement in ideal settings, difficult in other settings and effectively impossible in still others. Sometimes the difference between impossible and easy will be a matter of buying one copy of a relatively inexpensive piece of software. At other times, moving from impossible to easy would require creating a campus-wide infrastructure.

There are a lot of ways to cut up the testing process, but I think we'll do well to divide it into the five categories I've already alluded to: developing, administering, scoring, recording results, and providing feedback to students.

Test Development

Computers are really, really good at storing and retrieving information, particularly well-structured information, so it is not surprising that some of the earliest applications of desktop personal computers to testing took the form of "item banking" programs. Some of us bought such programs as much as a decade and a half ago and others of us were introduced to them by publishers when they began providing item banks on diskettes accompanying their textbooks. These early programs allowed us to print tests consisting of the items we selected, but they were frustrating to use because they only worked well with multiple-choice, text-only items and they weren't designed to store information about the items.

Today's item banking programs match our expectations that computers will be able to handle text, graphics, equations, and in the case of online administration, time-based (e.g., video or audio) items all with equal ease. Storing our items, of course, only makes sense if we plan to reuse them. If we plan to reuse items, we will want to store information about the items as well as the items themselves. For example, we might want to store the instructional objective the item addresses or the cognitive level at which we expect the item to function. Reflecting their designers' experience in large-scale testing programs, many item-banking programs also store item statistics (e.g., difficulty and discrimination indices), sometimes for each of several administrations of the item on different tests.

The tools we use naturally affect the way we work--sometimes for the better and sometimes for the worse. There is a risk to be managed in storing (and presumably using) item statistics in the typical instructional setting. If I find that a particular item is very difficult for my students this semester, I'm probably going to change my instruction to help more students master that material. Paradoxically, the more effective I become as a teacher, the less stable the item statistics will be.

Test Administration

Item banking is relevant no matter how a test is going to be administered. Online administration of tests, though, offers increased efficiency as well as making feasible options that are not usually feasible with paper-based testing. Some of the efficiency benefits fall under the headings of scoring, recording and providing feedback and I'll discuss those below. But, for me, the most exciting opportunities lie in a computer's ability to create richer, and sometimes more authentic, testing environments. My colleagues James Pusack and Sue Otto have created systems in which authentic video (e.g., soap operas and commercials) from Germany, Spain, and other countries form the basis of a wide variety of exercises. Video is still pretty challenging to deliver, but I know a lot of faculty members who are glad to be able to present reasonably high quality color still images as part of a test, and that's pretty straightforward. Building a test around something that can't be photocopied is, in many disciplines, a wonderful opportunity.

Test Scoring

Computers have been scoring tests for about fifty years using response forms that can be scanned by machine. At a minimum, online administration represents a convenience because forms don't have to be transported. In the better online testing systems, even more is possible. Scannable forms are effectively restricted to "selected-response" items (e.g., multiple choice, true/false, matching). Online testing allows students to type in responses. In the case of very simple "constructed-response" items, such as fill-in-the-blank items, the computer can do quite a good job of scoring. Even for more complex items (short answer or essay), many faculty are finding that grading typed responses online is much more efficient than grading traditional blue books. A lot of the efficiency comes simply from the fact that the responses are typed. In addition, though, some systems allow the grader to enter a comment along with the score for the item. In these cases, it is sometimes reasonable to reuse "boilerplate" comments, which can simply be pasted in.

Recording Test Results

With online testing, scoring and recording are not separate steps. As each item is scored, whether automatically or by hand, the

result is entered into a student database. When all the scoring is done, all of the recording is done as well. Not only are the individual students' results available, but summary statistics and graphs are available as well.

For a variety of reasons, many of us are not in a position to use online testing. Computers are still making our lives easier. I venture to guess that many of us are very glad for a spreadsheet's ability to store, organize, and summarize test scores. For those who don't find a spreadsheet congenial, easy-to-use grade book programs exist.

Providing Feedback

The final step in the testing process, as I've divided it up, is providing feedback to students. Again, online testing provides remarkable opportunities. Prompt feedback is one of the AAHE Seven Principles of Effective Undergraduate Education (Chickering & Gamson, 1987). For automatically scored items, instructors can provide item by item feedback or choose to provide feedback at the end of the test. It is possible to provide an explanation for each alternative the student might have chosen so that he or she knows not only whether the answer was correct or not but why. If the items can't be scored automatically, students can still get their results as soon as the scoring is complete instead of waiting for the next time the class meets.

Realities Of Online Testing

The Downside: Security

As you can see, I think that some of the most exciting opportunities for the use of computers in testing rely on online administration. Nevertheless, the first time I talked to a faculty member who was considering this option, I felt very nervous. The word "security" kept flashing (in large red letters) through my mind. I think we all want the score assigned to a student to represent that student's accomplishments on that test. Feeling comfortable about test security usually comes down to feeling comfortable that (a) the person whose name is associated with the test is indeed the person who took the test and (b) the students were not exposed to the items before taking the test. If that comfort isn't provided through an honor code, we have to establish it through our testing procedures.

If all of our students take the test online but at the same time in one or more proctored rooms, we can feel as secure as we do with traditional paper and pencil testing. On the other hand, if students take the test online at a time and place of their own choosing, we really have no control over the identity issue and only very modest control over the seeing-the-test-beforehand issue (through, for example, administering alternate forms).

The Upside: Time Saved

Even so, I have at least two colleagues who routinely give graded quizzes online and allow students to take the tests at the time (within a defined window) and place of their choosing. The reasoning of these colleagues is tightly tied to their particular situations. First of all, we have the necessary infrastructure here on campus. Second, each of these teachers teaches a very large (enrollments of 200 to 300) class. Finally, these weekly reading quizzes are very low stakes events accounting for perhaps 10% of the student's grade all together.

Now look at what they have gained. They can give a weekly quiz without giving up any class time to administer it. In classes this large even a very brief quiz consumes probably 15 or 20 minutes, so these teachers have gained literally hours of instructional time. In addition, once they have made up the quizzes, they spend no time scoring, recording, or providing feedback. That's all done automatically. Before walking into class, they know how many students completed the quiz (and therefore how many did at least some of the reading) and they know if there are any items that were missed by a large number of students, so they can plan their class time accordingly. Finally, students have already received their feedback on the quiz.

Is all of this worth the security risk? After all, some students may have given their passwords to a friend and had the friend take the quiz for them. Of course, if the test had been administered in an auditorium, some students might have cheated as well. We don't know which system would have more cheating. To my two colleagues, the benefits are well worth the security risks. They've found that using a computer to take over the parts of the testing process a computer can do well leaves them more time to devote to the parts of teaching that only a human can do well. Isn't that exactly what we hope for from learning technologies?

View the Virtual Companion at www.ntlf.com

Contact:
Tom Rocklin
Professor of Educational Psychology and
Director, Center for Teaching
4039 Main Library
The University of Iowa
Iowa City, IA 52242

Telephone: (319) 335-6048 Fax: (319) 335-6073
E-Mail: thomas-rocklin@uiowa.edu