 "Photo: Kara Baranski/Acting Design Editor")
After changing her major twice and taking 62 courses in 25 departments, recent University graduate Michele Bruder said she believes the survey forms on which students rate their teachers each term do serve a purpose.
"They make adorable pink-and-white paper airplanes," she said.
Bruder's observation reflects a widespread suspicion among students that filling in the circles and writing in the comment lines of the forms have little effect on teaching quality at the University.
The surveys should provide the University an opportunity to gauge students' opinion of their teachers, but does its handling of the results give students a good reason to be suspicious?
Analyze This
Despite processing 150,000 teacher-rating surveys each semester, the University's administrators opt not to analyze any of the resultant data.
The Teaching Excellence Center has been asked to analyze Universitywide survey data twice since the surveys were introduced in 1992, said Monica Devanas, director of faculty development and assessment programs. Both times the analysis involved student response rates, and both requests came from then-Vice President for Academic Affairs Joseph J. Seneca.
Robert Heffernan, director of institutional research for the University, said he has never been asked to look at survey data. "It does offer potential for analysis," he said, adding he would look at the data if Executive Vice President for Academic Affairs Philip Furmanski requested it.
The University's policy of using the surveys only to look at individual teachers concerns History Professor Rudolph Bell, president of the Rutgers Council of the American Association of University Professors.
Bell said there should be comparisons among departments, campuses, course levels and part-time lecturers versus full-time teachers, which could be used to improve the teaching process.
Furmanski, though, said he could not think of any useful information the University would gain from analyzing the students' ratings of the faculty. Such a pursuit, he said, would be a "silly" waste of money and time.
"I could spend my entire life looking at those 150,000 pieces of data," Furmanski said.
A Day in the Life of a
Rating Survey
The TEC, which distributes, collects and processes the rating surveys, has no system to ensure all teachers actually administer them - and many teachers do not.
Probably 75-80 percent of teachers participate in the survey process, Devanas said.
"People who are tenured and not up for promotion might not care," she said. "Part-time lecturers who teach one semester and are not coming back might not care."
After the TEC processes the survey forms, it sends them, along with a statistical summary sheet for each course section, to their respective department chairs.
Some chairs said they read every student comment when they review the survey results.
Writing Program director Kurt Spellmeyer said he spends a couple of weeks reading all the comments to see what students think. "That's part of my job, and I take it very seriously," he said.
The eight current and former department chairs who were interviewed for this story all said they review the survey results to identify and deal with teachers who received low scores. On the surveys' 1-to-5 scale, most said they consider a 3 to be a low score.
Identifying problematic teachers by looking for low survey scores sounds reasonable, except teachers almost never receive low scores. Of last spring's survey results for 2,268 Faculty of Arts and Sciences course sections, 3 percent showed a score of 3 or below for the question regarding teaching effectiveness.
Grade Inflation?
When asked about the surveys' persistently high scores, Furmanski said he hopes it reflects general student satisfaction with the education process and the "excellence of teaching" at the University.
Alfred Vogel, an English department lecturer with 30 years' experience in survey research, offered a different explanation for the high scores. He described a "halo effect" he found in organizational studies of companies, when employees were asked to rate their bosses. Employees almost uniformly gave their bosses high grades, probably because people tend to respect their bosses, he said.
"I would argue that teachers are to students as bosses are to employees," Vogel said. "In fact, when I first came to Rutgers and heard about these surveys, I said, 'I bet the ratings are high for most people.'"
Bell said high teacher-rating scores are not only a University phenomenon. "I think students nationwide are kind in ratings," he said.
A low student response rate is another likely factor affecting the survey scores. Michael Scriven, associate director of the evaluation center at Western Michigan University, said one frequent error associated with student ratings is failure to ensure an "acceptable return rate." He said he considers an acceptable rate to be more than 80 percent of students enrolled.
However, many departments featured return rates below that figure. For example, in the economics department in spring 2004, the average student response rate was 47 percent. In computer science, the average was 56 percent; in philosophy, 60 percent.
Gary A. Gigliotti, director of the TEC, pointed out the center lists class enrollment and number of responses on the statistical summary sheets. Department heads, not the center, should look at these numbers and address instances of low response rates, he said.
"If [department heads] are not using the information appropriately, that's their fault," Gigliotti said. "That's their job."
The Next Step
When the chairs finish reviewing the rating forms, each teacher is expected to read his or her results. However, department procedures for this vary greatly. Some departments send teachers the entire packet of original survey forms and the statistical summary sheet, and others send only the summary sheet.
Most teachers who were interviewed for this story said they thoroughly review their rating forms - particularly the student comments - and work to improve their teaching methods based on this feedback.
"We are all hungry to know how we've done," said Professor Jenny Mandelbaum, the communication department chair. "It's just like students getting a test back; they want to know what the professor thought."
Not all teachers regard the rating surveys as important.
Ellen Posner, who taught in the Writing Program in 1998-99, said she initially did not even read her survey results. "I don't see how they're helpful to anyone," she said.
Tenured history Professor Jack Cargill agreed. "I do not feel that the evaluations are ever very helpful or informative, since they are essentially a popularity contest," he said. "Students who like their grades give high evaluations. Students who do not like them give low evaluations."
Who's Afraid of a
Rating Survey?
Tenured professors not applying for promotion need not fear the teacher-rating surveys.
Such professors are evaluated by their department once every five years, during post-tenure performance review. They are also evaluated if they apply for a pay raise, known as merit pay, which they may do once a year. However, a review of rating scores is not required during either of these evaluations.
Associate Vice President for Academic Affairs Karen Stubaus said the purpose of post-tenure review is simply to ensure dialogue between chairs and faculty.
"It is not designed as a check on how well a professor is teaching," she said, and the use of rating scores during this process is therefore not mandated.
Professors who are not yet tenured or have not yet been promoted to the rank of full professor have only slightly more reason than full professors to worry about the rating surveys.
All professors are evaluated after teaching for three years. They are evaluated again when they apply for tenure, which they may do after teaching for six years, and again when they apply for promotion to a full professor, which they may do five years after receiving tenure.
During these evaluations, professors complete a form pertaining to their quality of teaching, research and service, in which they include their average rating scores from the questions regarding teaching effectiveness and course quality for every course they have taught since at least their previous evaluation.
"This Is a Research University"
So rating scores are considered during promotion and tenure decisions as one measure of a professor's teaching quality. However, do these decisions rest more on research quality than on teaching or service?
"This is a research university, and teaching has not held a primary position," Devanas said. "But I would like to see it elevated to be more in the consciousness of all faculty."
Geological sciences department chair Kenneth Miller agreed with the importance of research. "In most cases, 50 percent of the emphasis is put on research, 30 on teacher quality, and 20 on service," he said.
The University allows each department to set its own rehiring policies with regard to lecturers, PTLs and teaching assistants. The University does not require, or even recommend, rating scores be consulted for these rehiring decisions.
What Students Don't Know ...
The survey results are posted online at http://sirs.rutgers.edu. But do students know they can access the surveys online?
Devanas said the TEC has tried to inform students about the survey, but she estimated that only 10 percent of students know the ratings scores can be accessed online.
Even students who do access the rating survey Web site cannot easily use the data found there, because the site lists scores by course, semester and campus, not by teacher's name. Devanas said this glitch should be corrected with the posting of the fall 2004 scores.
************************** Some of the problems with the surveys are:
* The University has no system for analyzing the survey results and finding Universitywide trends in order to improve teaching methods.
* At least 20 percent of teachers do not participate in the survey process, an official at the University's Teaching Excellence Center said.
* In many classes, less than 60 percent of enrolled students hand in the surveys.
* The University does not require rating scores to be considered in post-tenure performance reviews.
* Professors' scores do factor in promotion and tenure decisions, but University officials say these decisions hinge more on research quality than on teaching quality.
* Department chairs are not required to consult survey results when rehiring lecturers, part-time lecturers and teaching assistants.
* The University posts survey results on the Internet, but a TEC official estimated only 10 percent of students know this.
Be the first to comment on this article!