Assessing Candidates

In a few days, our early-career-faculty (ECF) group is hosting a discussion for non-tenure-track faculty on the other side of the hiring process [1]. That is, they want to know a bit more about how departments evaluate candidates for tenure-track positions, and they’ve invited a variety of people who have chaired searches in to speak to them. Unfortunately, I had to decline [2] because I had a previously scheduled meeting at the same time. But we are midway through a search, and the College has asked that departments better document parts of the search process, so it seems reasonable for me to summarize some of my thoughts on the matter for sharing with the group.

The first thing I should note is that there are significant differences between disciplines and between departments. Some differences naturally arise from external factors; departments with thirty candidates will have different processes than departments with three-hundred candidates [3]. Some differences arise from departmental perspectives. For example, a few departments at the College, including CS, expect that a competent faculty member should be able to teach almost any course in the undergraduate curriculum [4]. Others expect faculty to focus their teaching more. Some do preliminary interviews at professional meetings; others do phone interviews. Some have uniform timetables nationally; some can vary significantly [5]. So, while I will describe what happens in Computer Science, I make no guarantees that it bears any resemblance to what happens in other departments at Grinnell, or in other Computer Science departments across the country.

We have a few broad criteria we use in assessing candidates, criteria that come into play in different ways at each stage of the process. These are (more or less):

• Teaching. Does the candidate appear to have potential to be a strong teacher, one who embraces new approaches and can readily employ active pedagogy?
• Scholarship. Does the candidate appear to have potential to be a strong scholar, one who regularly produces interesting peer-reviewed scholarship? Is the candidate likely to be able to involve students in scholarly projects?
• Diversity. Does the candidate show understanding [6] of the significant diversity issues that face our profession and show potential to help address those issues, at least at a local level?
• Institutional fit. Grinnell is an undergraduate-only liberal-arts institution that emphasizes close student-faculty work, including mentoring relationships. Does the candidate show understanding of that type of environment and a promise of success?
• Push [7]. Is this someone who has informed opinions about the issues above, and is willing to discuss and debate them?

Soon after the deadline, we begin by doing a first read of the applications. We try to make sure that each application is read by at least two, and preferably three, faculty members. At this point, we use a very coarse-grain scale: High (good chance for being good colleagues, according to the criteria above); low (significant deficiency in one or more areas); or middle (neither of the above). We then sit down and look through the ratings and get ready for the next step. We spend some time discussing the ratings, partially to help norm ourselves, partially to help newer faculty think through approaches (and, sometimes, to help inform the older faculty), partially to make sure that our Equity Advocate can better observe and respond to our process.

The end goal of this process is a group of candidates whose applications will be read by every member of the committee (usually every department member). That group of applications usually includes every application that has at least one high rating (even if it also has one low rating) and, depending on the size of that pool, some of the applications that have all middle ratings. Our target is usually about twenty applications, or perhaps a bit fewer, but if we see lots of high ratings, we’ll deal with more.

---

How do we evaluate according to the criteria in the first read, particularly given that we are often looking for potential, rather than just past accomplishments? The statements tell us a lot. Someone who has potential to be a strong teacher can think through teaching issues and experiences, even if they have not yet taught. What kinds of issues do they forefront? How do they draw upon their own experiences as a student. The research statement not only lets us know whether they do a kind of scholarship that would fit into our institution, but also whether they can present their ideas in a way that is accessible to non-experts. The diversity statement tells us whether or not they understand that our discipline has problems with severe underrepresentation by certain group and how they think about addressing that problem. Some candidates have already done things to help address the problem; others have read the literature and can speak to processes that seem to work; others have good intentions and inclinations.

As you might expect, the CV, reference letters, and transcript provide valuable additional data. I’ll say that there’s no particular thing we look for in either of those parts of the recommendation; just that they tend to inform us in various ways. For example, if we are unsure about the breadth of a candidate’s teaching range, we might look at what courses they’ve taken.

What gets you eliminated before the second read? A variety of things. As I noted, a significant deficiency in any of the key areas (although not yet for Push) may get you eliminated: a narrow teaching statement, an inadequate diversity statement; a difficult-to-understand research statement. Poor writing will eliminate some candidates: Faculty at Grinnell have to teach Tutorial; we want to be sure that our candidates can do that well. A misunderstanding of the type of institution can damage a candidate, such as when they write about projects they plan to do with graduate students. It’s rare, but I’ve seen a few cases in which a poor recommendation from the candidate’s advisor raises warning signs.

What makes you more likely to get a second read, other than just strong statements and good letters? Showing that you’ve paid attention to the institution and the department helps. (Please don’t just cut and paste a few lines from the department’s Web site; but do show that you’ve read it.) That may not be necessarily; in CS, strong statements will generally be all you need to get your application a second read.

---

Okay, back to the main process. We’ve completed the first read. We’ve identified candidates for a second read. What next? During the first and second reads, we try to put notes in a shared place [8] so that we can skim through the kinds of things our colleagues are thinking.

After everyone has read all of the applications identified for a second read, we sit down again as a department and talk through all of those candidates, identifying both strengths and weaknesses. Sometimes we put those on a whiteboard for everyone to see; sometimes we just talk them through and each take our own notes. At this point, we normally ask one more question: What new does this person bring to the department? We don’t want clones of ourselves [9], so we consider how this person will strengthen the department. Different members of the search committee are likely to identify different strengths. Some may note the value of intersections with another department, others may note the possibility of adding another perspective to a research group, others may note interesting outreach activities.

Then comes the hard part: Narrowing the pool. I can’t say that we’ve ever come up with an ideal approach. At this point, there are usually a few candidates who are clearly at the top of the list, as it were, a few candidates who are clearly at the bottom, and a bunch more in the middle. We’ve used a variety of strategies over the years, from rough consensus to quick everyone picks five as a way to more clearly see general trends. We don’t usually use that as the final arbiter, but it informs the followup discussions.

---

What gets you through the second read? That’s harder to say. It can be a statement that particularly stands out from the rest. It can be a particularly thoughtful cover letter that shows that you understand Grinnell. It can be something you’ve done elsewhere that we would love to add to the department. It could be that you’re in an area that we think would particularly strengthen the department.

---

We then phone interview all of the candidates. I know that the CS department’s interviews are different than most: We have a long series of questions to ask (longer than most) about teaching, scholarship, diversity, and more. However, we also feel free to deviate from that series of questions if the candidate asks about something or raises an issue we’d like to know more about. CS does phone interviews, rather than Skype (or other videoconferencing software), because we think it helps the candidate. For example, on a phone interview, we don’t see the candidate scrambling to look something up in their notes or on the Internet.

---

What makes for a successful phone interview? Much the same as what I noted makes for successful statements. We want to see [10] someone who is thoughtful about teaching, who can speak clearly about their scholarship, who can speak to diversity, and who can respond well to questions about all of these issues. We like to talk to candidates who have done their homework, as it were. This is also a time we look a bit toward issues of push - Is this someone who will participate well in our regular departmental discussions of these kinds of issues?

A few candidates tend to eliminate themselves at this point. They ask about graduate students. (Okay, that doesn’t completely eliminate them, but it makes us think they don’t really know what job they’ve applied for.) They seem uninterested in our questions, or in asking their own questions.

Tip: Have well-thought-out questions. They make a huge difference. If you can generate good questions on the fly, that’s also a plus. As I tell my students, asking good questions shows that you are interested in the position and engaged in the process, and therefore can make a huge difference, no matter what kind of job you’re applying for.

---

Back to the main process. We have another painful set of decisions to make. Grinnell usually lets us bring only three candidates to campus. So somehow we need to narrow the pool. This process usually takes a few meetings of a few hours each. We start by going through the strengths and weaknesses of each candidate. We give every member of the search committee a chance to speak so that we have a reasonably comprehensive list of those strengths and weaknesses. But at some point, we need to choose.

Unfortunately, it often seems that we are comparing apples to xylophones. That is, candidates show very different sets of strengths and weaknesses. We return once more to the question of what new each candidate would bring to the department. We ask if we agree on any particular sets of qualities that are more compelling. Rarely do we even try direct comparison on individual properties (e.g., Does candidate X have stronger teaching potential than candidate A?) However, we do sometimes try to compare to candidates and ask if there is consensus on which one is preferable.

I will note that I expect that the CS department will likely prioritize thoughtfulness about supporting diversity at this point, but I can’t be sure.

And, once again, we sometimes inform the process by asking members of the search committee to indicate their top three candidates. That often tells us where we have and lack consensus, and which candidates we may need to compare.

There are a number of issues we don’t consider at this point (or, really, at any point). We don’t ask ourselves whether the candidate would be likely to come to Grinnell. We don’t consider family issues. (If we know that the candidate has a partner, and we’re planning to invite the candidate to campus, we do try to identify potential opportunities for the partner.)

---

I have no good suggestions for what will help you get to the top of the list at this stage, other than what I noted in the phone interview.

---

Okay, we’ve reached the on-campus interview. Candidates meet with all of the department faculty, with administrators and faculty leaders, with faculty from other departments, with students, and more. We ask them to teach a class and to give a talk aimed at upper-level undergraduates. (From what I can tell, some departments at Grinnell are discontinuing the teach a class component of the on-campus interview.) They have dinner with the department. (Other departments may do a cocktail party or something similar.)

---

What makes a successful on-campus interview? In part, you have to be on as much of the time as you can, no matter who you are talking to. The department gathers information from almost everyone the candidate encounters, unless we’ve promised otherwise (e.g., we are looking for ways for candidates to talk about issues that are important to them, but that they don’t want the department to know about). And we do care about how the candidate treats people; I don’t think I’ve seen it in any of our candidates, but I wouldn’t hire someone who treats our academic support assistants (or any staff member) poorly.

In the sample class, we look to how you address the subject matter (e.g, are you just lecturing on a section of the textbook, or have you come up with your own approach and some active learning exercises) and how you interact with the students (e.g., how do you deal with questions; do you treat students of different genders and races differently). For your lecture, we again look at your ability to explain your work to a non-expert audience. We also see how well you answer questions and how you treat the members of the audience. So consider those issues.

What else? As in the phone interview, work to ask good questions and to answer questions well.

---

Now the department is left with the last decision. And it is usually both an easy and a very difficult decision. Most of the time, all three candidates would be great additions to the department, but we have to identify the order in which they will be asked [11].

We gather information from almost everyone who has met the candidate. Council members send us reports. Students send us email messages, catch us in the hallway, or relay their opinions through the Student Educational Policy Committee. Colleagues in other departments may chat with us or drop us a note. (Yes, I was serious that you have to do your best to be on for all of these folks.)

How does the department prioritize? Once again, we discuss each candidate, making a list of strengths and weaknesses as we go. We consider what new each candidate brings to the department. We try to reach consensus, but it’s hard, since we are rarely in perfect agreement. (I recall one recent search in which another faculty member and I ranked the candidates in exactly the opposite order.) We use a variety of techniques to reach consensus, including, as the prior example suggests, individually ranking the candidates and sharing those rankings with each other. We don’t use those individual rankings as a final step; rather, they are a way to stimulate more discussion.

It’s also usually a long process; the department has to debate (and debate, and debate). Then we write a proposal to the Dean and Executive Council. Then they have to decide. For whatever reason, somewhere during that time we have a Holiday, and everything falls behind another week or two. So we sometimes have to deal with questions from candidates, and can honestly say The process has been slower than we would have hoped.

But at some point, the Dean tenders an offer and negotiates with the candidate (or candidates). Then we’re done, or at least we hope we’re done.

---

Yes, there may be a bit more after this point, but I don’t think it’s sufficiently informative to add anything else.

---

[1] No, they don’t call it that. It’s just what I think of it as.

[2] Given my tendency to ramble, it may be fortunate for the ECF group that I’m unable to attend.

[3] CS normally gets eighty to ninety candidates for a tenure-track position, but we see some significant variation.

[4] We don’t actually require faculty to teach every course in the curriculum (although I think Henry Walker and John Stone may have done so). And, even though we want faculty to be able to teach a wide range, we know that both faculty and students are better served by faculty teaching a smaller range over a few year period.

[5] In CS, I know that most of the R1 institutions have deadlines in early spring, while many liberal arts colleges have deadlines in mid fall.

[6] You’ll note that this is one time I didn’t say potential. That’s because we expect that anyone who would care about diversity would already have observed the problems in our discipline.

[7] The term Push is due to my colleague, John Stone.

[8] Usually a shared physical place; as computer scientists, we don’t particularly trust anywhere online to be appropriate.

[9] CS students may laugh at this, but it’s true.

[10] hear, to be more precise.

[11] I keep trying to convince the dean that it’s okay to make multiple simultaneous offers, not only because most candidates in CS have multiple offers, but because we could use more than one new faculty member. But that argument never seems to work.

---

Version 1.0.1 of 2016-11-06.