Developing an alternate end-of-course evaluation

Topics/tags: Grinnell, assessment

Grinnell, like many institutions, asks students to evaluate each course at the end of the semester. End-of-course evaluations (EOCEs) traditionally have two non-compatible goals: Faculty members use them to figure out how to improve their classes and institutions use them to evaluate faculty. Grinnell’s are used to evaluate faculty (more on that in another musing) and so provide less utility in improving classes.

During my first few years at Grinnell, the faculty voted to institute a common form for EOCEs; prior to that, each department used their own form [1]. I think that’s a point that they became more a tool for faculty assessment than for pedagogical improvement.

Unlikely some institutions, that switched to electronic EOCEs, we still use paper forms. I’m pretty sure that’s a good thing [2]. For example, from what I understand from the popular academic press, students become even more biased against women and people of color when filling out online evaluations [3].

The content of the EOCEs we use now is essentially identical to that of the EOCEs we developed twenty years ago. They ask six Likert-style questions [6,7].

1. The course sessions were conducted in a manner that helped me to understand the subject matter of the course.

2. The instructor helped me to understand the subject matter of the course.

3. Work completed with and/or discussions with other students in this course helped me to understand the subject matter of the course.

4. The oral and written work, tests, and/or other assignments helped me to understand the subject matter of the course.

5. Required readings or other course materials helped me to understand the subject matter of the course.

6. I learned a lot in this course.

Thanks to a suggestion by Lee Sharpe, they also provide room for students to comment. Those comments are useful. For example, we do sometimes get comments in which students strongly disagree with I learned a lot in this course and then add a comment like Best course ever! We generally assume that learning little is not one of our student goals.

In any case, it’s been nearly twenty years since we developed the form. Is it perhaps time to develop a different one?

One thing that got me thinking about this issue was a recent presentation about one of the important co-curricular activities on campus. In exploring the value of those activities, the designers of the survey asked questions about the College-wide learning outcomes. Their slides said things like

82% agree or strongly agree that this activity has helped them develop the ability to communicate clearly and effectively

I was happy to see that they reported categorical data in a sensible way. It’s much better than what the institution does with the EOCE data [8].

I decided to conduct an experiment, mostly for my own purposes. This semester, when I distributed the official forms, I’m also distributed some unofficial forms. These unofficial forms replace the current Likert-style statements with the following [9].

1. This course helped me develop creative thinking skills.

2. This course helped me develop critical thinking skills.

3. This course helped develop my ability to communicate clearly and effectively.

4. This course helped me develop a sense of social responsibility and fairness.
   

5. This course helped me develop the ability to continue to learn individually.

6. This course helped me develop the ability to continue to learn collaboratively.

7. This course helped me develop the ability to approach a question from multiple perspectives.

As you may note, I separated outcome 1 [11] into two statements and outcome 4 [12] into two statements. I separated #4 following the lead of the survey report I saw. I separated #1 following a suggestion from a colleague. I did not add a question for outcome 6 [14], since it did not seem appropriate to an individual course [15].

When I first started working on this idea a week ago, I reflected on what other questions that I might add. There are two questions that I sometimes ask.

1. What is something you would recommend that we change about the course?

2. What is something that worked well and that we should keep doing in this course?

There are also two questions that Jerod normally asks.

1. The one thing that most helped my learning was …

2. One thing that would have helped my learning more is …

I ended up using Jerod’s questions rather than mine. I’m not yet sure that that was the best decision.

I was also tempted to ask how many hours they spent on the course. But that’s a number that likely varies from week to week, even though I try to keep the workload relatively consistent. I’ve also been told that students are also not the best estimators [16]. So although I left a question free for that question, I decided not to include it.

What will I learn from all this? We’ll see. I’ll report back on my results in a few weeks, after I’ve turned in grades and received my official and unofficial EOCEs [21].

Postscript: About fifteen years ago, I wrote a long essay for students about end-of-course evaluations. I’ll almost certainly rewrite that essay, not least because it is no longer completely accurate. Stay tuned.

[1] I always remind people that Math/CS/Stats, the department that most reflected knowledge of quantitative-reasoning, asked primarily qualitative questions; the one quantitative question was About how many hours per week did you spend on this class?

[2] I realize that not all students can read or fill out a paper form. We do provide an electronic alternative for students who need those kinds of accommodations.

[3] My comment about online EOCEs ended up at OASIR [4]

[4] Office of Analytic Support and Institutional Research. It was once just the Office of Institutional Research. When Kington arrived, he wanted to rename it to the Office of Analytic Support. Then he learned that we’re required to have an Office of Institutional Research. So we got the fun hybrid name, Oh, Sir [5].

[5] That story may or may not be true. It’s still a good story.

[6] When we evaluate faculty, we tend to look at questions 2 and 6.

[7] A Likert-style question is one in which students respond to statements with degrees of agreement or disagreement. On these forms, I think it’s Strongly Agree, Moderately Agree, Slightly Agree, Slightly Disagree, Moderately Disagree, and Strongly Disagree.

[8] I still don’t understand why people consider it appropriate to assign numbers to the agreement and disagreement and then do things like average them. Is Strongly Agree really twice as good as Slightly Disagree?

[9] In case I wasn’t clear, the new statements are also Likert-style statements. I even went so far as to paste [10] them over the original statements.

[10] Okay, I taped them over the original statements. And I did it in the physical world, rather than the digital world. The result isn’t perfect, but it’s acceptable.

[11] Students develop creative and critical thinking skills that allow them to analyze the work of others, formulate relevant questions, and respond to those questions in a substantive way using quantitative and qualitative evidence. https://www.grinnell.edu/academics/centers/ctla/college-wide.

[12] Students develop the ability to continue learning independently and collaboratively. https://www.grinnell.edu/academics/centers/ctla/college-wide.

[14] Students pursue a chosen field of study in depth and develop understanding of a core body of knowledge in that field as well as the ability to employ modes of inquiry appropriate to that field. https://www.grinnell.edu/academics/centers/ctla/college-wide.

[15] It may also be the case that not every Grinnell course addresses the other four outcomes. Nonetheless, I thought it valuable to ask about them. And, as I reflect on the courses I teach, I really do try to achieve each of the first five outcomes.

[16] One of my favorite EOCE stories has to do with the alternate EOCEs that we used to use for Tutorial [17]. That form used to ask How many papers did you write for Tutorial this semester? They stopped using it when they discovered that the number ranged from three to nine for the same Tutorial [19].

[17] That’s the real Tutorial, not this strange other thing from the IGE [18] that co-opts the term.

[18] TLA for Institute for Global Engagement.

[19] Yes, I realize that I’ve told that story before. I’ve reached the age in which I am likely to repeat stories again and again and again [20].

[20] I’m pretty sure that I’ve made a similar statement about my age in the past, too. I’ve also reached the age in which I remember less, particularly about what I’ve put in my musings.

[21] I made the new EOCEs without a class name because I’ll know what class it is. Now I’m trying to figure out how I’ll tell the CSC 322 evaluations from the CSC 151 evaluations. Let’s hope that the students wrote something so that I can tell them apart!

Version 1.0 of 2018-05-16.