Compiled by Tom Rocklin.

This month's Virtual Companion is designed to accompany the article Computers and Testing in The National Teaching and Learning Forum, Volume 8 Number 5.

So, you think you might want a computer to help you with your testing? You are going to have to either build or buy some software. I can't claim to have surveyed the entire universe of testing software, but I've looked at some of the products available. There are other products out there, but these are the ones I've at least heard something about. I'd like to do two things here:

1. Point out some of the products available (without providing anything like a real review of them) and

2. Share a laundry list of evaluation criteria (which I've cribbed, with permission, from a University of Iowa committee).

The Products

I've divided the available software into five categories, with at least one example for each category.

Online instructional packages which include testing components.

In this category, I've included products designed to support an entire online instructional environment. Testing is only one part of these systems.

The product with which I am most familiar is WebCT (http://www.webct.com/). The testing function is extraordinarily feature laden.

CourseInfo (http://product.blackboard.net/courseinfo/) is a very similar package.

Standard wisdom among the folks I talk with is that WebCT has a somewhat better feature set and CourseInfo has a somewhat better interface. Demos are available for both, so you can make your own decision.

Adopting either of these really makes sense if you want their other features. They are much, much more than testing systems.

High-end testing systems.

In this category, I place two professional-level programs. These are up to the task of developing and administering licensure and certification tests, so they are probably overkill for classroom testing. Still, if you or your institution has broad needs, these deserve a look.

CAT (http://www.catinc.com/products.asp) publishes a set of complementary programs for test devleopment and administration. The programs support a host of advanced psychometric features (including a rich range of item response theory modeling options).

Performance Evaluation Technology (http://www.tdcnet.com/) publishes PET 3.1 which is in many ways comparable to CAT's programs.

Mid-level testing programs.

Question Mark Corporation (http://www.questionmark.com/) publishes Question Mark which allows for item banking, test development, and administration via floppy disk, hard drive, LAN, or on paper. They also publish Question Mark Perception for Web delivery. This is a full featured system, but without some of the psychometric sophistication of the high-end systems.

Logic Extension Resources (http://www.lxrtest.com/) publishes a series of products ranging from the Personal Edition of LXRoTest to the Professional Edition. These provide much the same sort of capabilities as the Question Mark products.

Personal level testing programs.

Scantron (http://www.scantron.com/) makes more than machines to read answer sheets. Among their other products are ParTest for authoring and item banking and ParScore for scoring, analysis, and gradebook functions. These products are oriented toward an individual instructor and printed tests.

Online testing CGIs.

If all you want to do is administer online tests and quizzes, you might want to look at these CGIs. A Common Gateway Interface is simply a program that works with a Web interface. If you have some technical skills (usually knowing some Unix is helpful), you can install these on a web server and take off. The ones listed below appear to be available without cost, and if you speak Perl or C you could modify them to your own specifications.

Learning Space (http://weber.u.washington.edu/~lspace/)

PHATE (http://www.cclabs.missouri.edu/~phate/) (particularly appropriate for math-based disciplines)

WebQuiz and WebTest (http://www.npcts.edu/acad/physics/WebQuiz/)

WebTest (http://fpg.uwaterloo.ca:80/WEBTEST/) (same name, different program)

TestProc http://fpg.uwaterloo.ca/testproc/introducing_testproc.html

Quiz Engine (http://www.echo-on.net/bytes/quiz.html)

The Criteria

I've pointed you in the direction of a lot of products. As you start looking at them some evaluation criteria would probably be helpful.

The following criteria were developed by a committee at The University of Iowa charged with identifying an all-purpose testing system to support everything from single-instructor pencil-and-paper testing to large-scale online testing. (For what it is worth, the committee concluded that no product available today would meet all of those needs).

If you go through the criteria and sort them into those that are essential to you, those that are desirable, but not essential, and those that you don't need at all, you will have a good scoresheet for comparing products.

Administrator interface:

• Provide a friendly graphical user interface (GUI) that is easy to learn and use.

• Perform free text searches within categories.

• Communication with other software products through text files and conversion utilities. For example, import Exam Service test item performance data matching it to the appropriate question in the bank of questions.

• Spelling and grammar checking with optional medical and legal dictionaries.

Test items:

• Create questions in multiple formats: multiple choice, true/false, extended matching, short answer/fill in the blank(s), etc.

• Utilize common word processing conventions: Cut and paste, font modifications (italics, bold, subscript, superscript), Greek and Latin symbols, foreign language fonts, and mathematical symbols.

• Questions can include graphics, digitized images, audio clips, animations, and movie clips.

• Each question has fields for storing question type, author, content topic, date authored, date last used in a test, learning objective, learning response, cognitive level--for example, Bloom's taxonomy, instructor feedback which could be printed off following paper and pencil exams or be available as a learning response in the test delivery mode, etc.

• Organize questions by topic and subtopic headings or according to objectives.

• Each item can have 7 or more multi-dimensional categories for storing preformance statistics such as difficulty and discrimination.

Tests:

• Test preparation allows for instruction selection and random selection using criteria based on the fields within a question.

• Create multiple forms by rearranging questions (either randomly or by chosen design); keys reflecting the multiple forms and rearrangements would accompany each form. This might require a scoring module.

• Provide predictive statistics when developing a test.

• Use Item Response Theory (IRT) statistics as an educational tool for instructors. Provide help to the instructor to interpret the statistics, possibly a tutorial.

• Educate the instructor on how to construct *good* tests.

• Build tests based on multiple criteria (objectives, keywords, content, statistical values, etc.), by random sampling or selecting item-by-item, and from multiple test banks.

• Ability to randomize the order of choices for an item.

• Allow multiple correct answers with different weights for each answer.

• Correction for guessing.

Test banks:

• Create multiple banks of questions with unlimited number of items in each bank.

• Import existing questions and corresponding data from existing test management systems.

• Integrate with the test delivery program.

Other:

• Support optical scoring of test items so that feedback on student and class performance can be provided to the instructor and an item analysis for each question can be imported directly to the question database to update appropriate fields.

• Available on more than one platform or in a client/server environment with client software on multiple platforms.

• Guarantee security, if on a stand-alone or on a network via a password.

• Produce a hard copy camera-ready exam utilizing a preferred typeface and font size.

• Adjust pagination to avoid splitting a question between pages.

User interface:

• Present graphics, digitized images, sound and movie clips embedded within questions.

• Present feedback or learning response to explain correct answer.

• Provide a note function for students to ask about particular questions or to identify errors within the bank. Could be set up as e-mail.

Test items:

• Track item statistics: test takers score, percentage, start time and date, elapsed time, item response data, ...

Tests:

• Work in tandem with the test management system.

• Deliver the questions from test banks in a random or predetermined order.

• Grading capabilities:
  1. Provide test performance report of individual examinee. % correct - relative rank.

  2. Provide individual examinee response summary by item. Error report - recorded of response vs. correct response.

  3. Provide class test performance report. Distributions - SDs - Means - reliability - discrimination.

  4. Provide item statistics - item analysis - discrim. - difficulty - theta level

  5. Ability to customize report data from 1-4 to include only the desired info.

• Ability to deliver computer adaptive testing using (Item Response Theory (IRT) statistics.

• Provide a better educational experience for the student by, for an example, interpreting test performance statistics so that more informative feedback could be given. One example of such feedback might be how *well* the student met the learning objectives for the test.

• Deliver test items over a network or though a web site.

• Provide access only via passwords or registered IP addresses.

• Guarantee security if delivered from a stand-alone, over a network, or from a web site. Has security for test material, student tracking information, html source code, and ensuring that only registered students take the test.

• Allow retries

• Allow timed tests

• Allow tests to be included within a tutorial

Other criteria

• Viability of the company/vendor.

• Other services offered by the company.

• Option to purchase source code.

• Support issues: Consider what the vendor offers. Consider the type of support the U of I - ITS, Exam Service, Center for Teaching, a department, etc., would need to provide in order to use a product. Example: Working through the "bugs" /quirks of a product (who?). Measurement issues (OCRME, Exam Service).

• Hardware requirements.

Acknowledgement: The University of Iowa committee that developed these criteria included Rick Dehn, Physicians Assistant Program; Scott Elliott, Office of Consultation and Research in Medical Education; Les Finken, ITS-Academic Technologies; Clare Kreiter, Office of Consultation and Research in Medical Education; Pamela Micheal-Milder, College of Nursing; Cliff Missen, Physiology; Jim Quigley, Evaluation and Examination Service; Robin Torner, Department of Pathology