After reading this morning’s nice piece at Inside Higher Ed on my colleague Yves, I did some catching up on old articles on the site. I found this post from a month ago: Hotness and Quality, another essay about the Rate my Professors (RMP) phenomenon:
If you’re not sexy, you might want to be easy.
At least if you’re a professor concerned about your rating on RateMyProfessors.com. James Felton, a professor of finance and law at Central Michigan University, and colleagues looked at ratings for nearly 7,000 faculty members from 370 institutions in the United States and Canada, and his verdict is: the hotter and easier professors are, the more likely they’ll get rated as a good teacher.
As far as students — or whoever is rating professors on the open Rate My Professor site — are concerned, nothing predicts a quality instructor like hotness.
Felton found a positive correlation of 0.64 (0.00 means there is no correlaton whatsoever, and 1.0 describes a perfectly linear relationship) between the “hotness†and “quality†— quality is a composite of “helpfulness†and “clarity†— ratings on the site. “Hotness†is determined by evaluators choosing “hot†or “not hot,†with each click counting as either +1 or -1. “Quality†is on a simple 1-5 scale. (Felton may be an exception on the correlation — while he doesn’t get any hotness points from RateMyProfessors, he does well on quality.)
Well, that’s moderately interesting. It’s clear that the ratings are hardly reliable, particularly considering that anyone — including other faculty members with an axe to grind — can rate professors. There doesn’t seem to be any reliable way of preventing the system from being totally abused.
Here’s what’s more disturbing: the article suggests that RMP could be used by hiring committees to check up on applicants. Suppose someone who is an adjunct prof at one or more colleges is applying for a full-time position. Isn’t it quite possible that a member of the hiring committee might let what he or she reads at RMP influence his or her feelings towards the applicant? I don’t know what percentage of the profs rated at RMP are adjuncts, but I would suspect that at least a healthy percentage are not full-time employees — and that many of those would someday like full-time posts.
I haven’t sat on a hiring committee in a few years. The last time was in 2000, before the advent of RMP. Applicants with teaching experience were allowed to submit sample evaluations from their classes if they chose; most did so. Obviously, these were on paper rather than on the ‘net, and the applicant controlled which evaluations the hiring committee got to see. Naturally, they excluded hostile or unpleasant or irrelevant comments. I can tell you that while they weren’t a decisive factor, these evaluations played a small part in our decision-making.
I’ll probably be on a couple of hiring committees next year. I suspect I will be able to resist the temptation to "check up" on the ratings of those adjuncts who will inevitably apply for tenure-track positions. But I wonder if my colleagues will be similarly restrained. This prospect genuinely concerns me. Perhaps hiring committees will have to take oaths similar to those taken by jurors — the sort where jurors are charged not to do any independent investigations while they are impaneled.
Damn. I never even thought of this. I generally get good course evals, but my my RPM contains four entries, two of which are negative. I’ve been in this profession for nine years, working hard on my teaching and writing. The idea that my hirability might be increased if I created a bunch of dummy accounts to give myself fake nice ratings is rather depressing.
On the other hand, your post inspired my to look myself up on that site, and the fourth (positive) review that made an appearance this term says I’m “hot” which is certainly a first. But the site is so obviously not to be taken seriously. Surely hiring committees can see that.
I am sure most folks on hiring committees wouldn’t take it seriously — but as the essay at IHE points out, it’s easy for one comment to get stuck in the head…
RMP should never be taken seriously by anyone, especially a hiring comittee. Professors on RMP appear to be rated almost entirely based upon how easy they grade their students and how good they look. “Lookism” is already rampant in society so I fail to see why we should keep reinforcing it. A good-looking person, on average, has more luck in pretty much every avenue of life than someone who is less attractive. I’m sure the hiring process for universities reflects this. My recollection of undergrad was that most of the younger, untenured professors were fairly attractive people and I was attending a fairly elite liberal-arts college. It may have just been my imagination but I have a hard time believing that a person’s appearance and attractiveness didn’t come into play somewhere in the hiring process.
Being an easy grader is the sort of information that gets passed around the student body on its own. In my experience the hardest semester of undergrad was my first semester or freshman year. I had no idea which professors were generally considered easier than others, so I chose classes based on nothing more than the course catalogue. I wound up taking four very difficult classes. After that I learned which professors were offering interesting courses that were also easy to get a “gentleman’s B” in, or which professors in my major would be more interesting or easier to work with regardless of the difficulty of the course. It was invalueable info but aside from my first semester it was the sort of thing that everyone just sort of knew by word-of-mouth. Admittedly, this could be very useful at the larger universities. Then again, since you’re likely to be taught by a TA as an undergrad at a larger university perhaps not. Either way, it’s not the sort of thing I would expect a hiring comittee to take seriously.
As a few around here might already know, I’ve been a grad student for a few years now, and teaching ratings are on my mind for reasons of both personal (I want to be a good teacher) and career (even the sorts of mid-level research departments I’m hoping to end up at are going to want someone who can keep first-year undergrads awake) development. Clearly RMP and similar websites are not the place to go for useful feedback; but I don’t think either of the traditional methods are all that useful either.
The first method I have in mind is the most common: at the end of the semester, students fill out a survey about their course and instructor(s). These surveys usually have a scantron component (‘How enthusiastic was the instructor about the material?’ Fill in circle numbered one through five.) and written response sections. My first problem is that the students rarely give useful feedback — if they write anything at all — in the written response sections. I suspect this is a combination of indifference and not actually knowing enough about good teaching to give the instructor advice on how to improve. The second problem is that the qualitative data is only useful after some aggregated analysis (‘Over the past four semesters, 72% of your students have rated you 3 or below in enthusiasm.’)– and at my old institution, I was simply handed an envelope with the scantron sheets, not analysis included. And my third problem is that these surveys are only slightly better than an independent website at gathering objective data — while a colleague with a grudge won’t be able to fill out a phony survey, what’s to stop a student who blames me for their 48% on the midterm?
The second method, having a more experienced instructor sit in on a few class sessions, is much less common, for obvious reasons. This feedback is certainly much more useful than what we get from the students, but it’s too limited. It’s nice to have some tips on how to lecture or run a discussion, but that doesn’t tell me how to write a solid exam or paper topic, much less a syllabus. These are all skills I’ve had to pick up on my own.
As a TA, it would be nice to have someone sit in on my classes a few times throughout the semester and looking over my syllabus, exams, etc. But as a future professor, I realize this would eat up a lot of some very busy people’s time.
I suppose teaching is like intelligence: there are a few people who are absolutely convinced that you can measure it quantitatively, and trying to do so mostly seems to work out to just wasting everyone’s time.
I agree that RMP isn’t the best place to get information on a candidate …. but, what about a situation in which the committee member looks at RMP after the interview and sees a strong pattern of things like, “ok teacher but really boring” or “says weird sexual things in class” — over many ratings — and those things confirm the committee member’s intuitions about the instructor?
I don’t think RMP is a great source for original information, but if 30 or 40 students over the course of many semesters report the same basic comments it should mean something, no?
PF, as far as I know, one disgruntled student could post 30-40 negative comments merely by using an easy IP proxy. On the other hand, a professor could rate himself or herself over and over again. (Something I was recently accused of, jokingly, by a colleague!) I’m told that the only thing RMP checks is whether or not the same IP address is used — you don’t even have to log in to rate.
I just don’t trust it much, but I know students use it. The concern here is the potential that it could be used to hurt vulnerable adjuncts and the untenured.
Interesting – I had two ratings, both horrible, when I just got back evaluations for the same class that were all excellent except for a couple middling. This suggests that the people who rated me on RMP didn’t even stay until the drop deadline.
RMP is not sending my password fast enough– i just have to know what the rest of the reviews are for mr. magloe.
“I need my RMP”: my friend, you are proving my point. Whatever you find there will be worthless, I’m confident.
This may sound like a quibble, but the strong correlation implies nothing about which direction the prejudice is flowing in. By which I mean: Maybe quality professors *seem* hotter to their students because they’re such good teachers. My personal gut feeling is that that particular correlation works in both directions. Hot professors seem better teachers than they are, excellent professers seem hotter teachers than they are.
Although I do agree that rating professors on their hotness is fundamentally stupid.