|
|
|
Test Your Assumptions about Student Evaluations: A Pop Quiz
| T | F | | 1. More difficult courses, with a greater workload, receive slightly higher student evaluations than do easier/lower workload courses. |
| T | F | | 2. Male instructors generally receive higher evaluations than do female instructors. |
| T | F | | 3. In general, age and years of teaching are moderately, significantly, and negatively related to student evaluations. |
| T | F | | 4. Research productivity is negatively related to student ratings of teaching. |
| T | F | | 5. There is a very small negative relationship between class size and student evaluations. |
| T | F | | 6. The academic field or discipline is unrelated to student evaluations. |
| T | F | | 7. Generally, there is a small positive correlation between the expected grade in the class and student evaluations. |
| T | F | | 8. Signed ratings are more positive than anonymous ratings. |
| T | F | | 9. Research has found a slight same-gender preference: male students give higher ratings to male instructors, and female students give higher ratings to female instructors. |
| T | F | | 10. Instructors' warmth and enthusiasm are generally unrelated to ratings of teaching competence. |
| T | F | | 11. Ratings in lower-level courses tend to be slightly higher than ratings in upper-level courses. |
| T | F | | 12. The scores on many specific items provide better data for personnel (summative) decisions than do a couple of global items. |
| T | F | | 13. There are high positive correlations between student and alumni ratings of global teacher competence. |
| T | F | | 14. Student evaluations are better measures of satisfaction with, and the effectiveness of, the course than the instructor. |
| T | F | | 15. There are moderate positive relationships between student achievement and student ratings of several instructor skills. |
Answer key
1)T; 2)F; 3)F; 4)F; 5)T; 6)F; 7)T; 8)T; 9)T; 10)F; 11)F; 12)F; 13)T; 14)F; 15)T
What Do Student Ratings Mean?
Kathleen McKinney, Coordinator
Center for the Advancement of Teaching
Illinois State University
How did you do on the true-false quiz about student evaluations? A great deal of research has been conducted over the last three decades on student ratings of teaching and of courses. The empirical findings from this research do not always conform to faculty members' beliefs.
Much of the research focuses on correlates of student ratings of courses or instructors.
In addition, studies have assessed factors that account for the most variance in student ratings. The literature also contains discussion of bias in evaluations (what is bias and when should we control for it).
Finally, writers in this area discuss the practical implications of their research.
Factors that are significantly and positively associated with student ratings include the following: measures of student achievement; alumni, peer and administrative ratings; qualitative student comments; workload/difficulty level; energy and enthusiasm of the teacher; status as a regular faculty member (as opposed to a graduate assistant); faculty research productivity; student motivation; student expected grade; and course level. The size and practical significance of these relationships vary. For example, most agree that there is little practical significance to the small positive correlation between expected grade and student ratings, and between faculty research productivity and student ratings. Similarly, research shows a small and negative, but practically insignificant, relationship between class size and student ratings.
Factors generally found to be unrelated to student ratings include faculty age and teaching experience, instructor's gender, most faculty personality traits, student's age, class level of student, student's GPA, student's personality, and student's gender (with the exception of a slight preference for same-sex instructors).
A variety of studies using several different methodologies have assessed which factors account for the most variance in student ratings. This research shows that it is the instructor and instructor variables,
Instructors may believe that student evaluations are unreliable. In general, the research does not support this belief.
rather than course variables, that account for the most variance in student ratings. Researchers conclude that most student evaluations should be used to evaluate instructors, not courses.
Finally, what is bias in student evaluations? When should we be concerned with bias? Experts in this area do not agree. Some have suggested that anything not under the control of the teacher constitutes bias. Others have posited that all variables unrelated to teaching effectiveness be considered bias. Finally, in some writings the focus is on variables that correlate with student ratings but are unrelated to teaching effectiveness and student learning.
For example, one might argue that faculty enthusiasm is a form of bias because it is part of his or her personality (i.e., not under the faculty member's control). Yet, one could also argue that this variable does affect teaching effectiveness and is, in part, under the faculty member's control--so it isn't bias. The issue of bias is still under debate.
What does this research tell us about using student ratings?
- Research indicates that the use of a small number (1-3) of global items is best (most reliable and valid for the purpose) for summative and personnel decisions, especially when comparisons will be made.
- To increase comparability across faculty and departments, evaluations should be administered with a standard set of institution-wide procedures.
- Research indicates that numerous specific items are best used for the improvement of instructors' teaching skills.
- Student evaluations should, generally, not be used to draw conclusions about a course. Research indicates that most of the variance in ratings is accounted for by the instructor, not the course.
- Though reliable and valid in many ways, student evaluations are still only one source of data on teaching. A multimeasure, multimethod approach should be used in evaluating teaching. Depending on the purpose of the evaluation, consider the use of peer observations/ratings, alumni ratings, teaching portfolios, review of teaching materials, measures of outside-class involvement with students, and measures of students' learning.
- Due to differences in student ratings by academic discipline, administrators comparing ratings for personnel decisions or awards across such disciplines should be very cautious.
- For formative purposes, collect student evaluations throughout the semester that are only for the instructor to see. Discussing these results with a colleague or faculty development person, however, is more likely to lead to improvement of teaching.
- Ratings of a course based on five or fewer student responses are of questionable reliability and ratings from an instructor's other courses should be used. Evaluation judgments about a faculty member are best made using ratings from at least five courses with at least 15 student responses in each course.
Instructors may believe that student course/teacher evaluations are unreliable, invalid, biased, and affected by many inappropriate factors. In general, the research does not support these beliefs. It is still the case, however, that student ratings, and their interpretation, have limitations. Student ratings should be only one source of information used for the evaluation of teaching, and the interpretation and proper uses of student ratings are at least as important as the ratings themselves.
(Sources: Braskamp, L.A., and Ory, J.C. 1994. Assessing faculty work. Jossey-Bass.; Cashin, W.E. 1995. "Student ratings of teaching: The research revisited." Idea Paper #32. Center for Faculty Evaluation and Development, Kansas State University.)
Contact:
Kathleen McKinney, Ph.D.
Center for the Advancement of Teaching
Illinois State University
301 S. Main, Room 107
Campus Box 6370
Normal, IL 61790-6370
Telephone: (309) 438-3694
E-mail: CAT@ilstu.edu
Reprinted by permission from the CATalyst (July, 1997).
|
