Meta-reviewing

Topics/tags: Overcommitment, writing, academia

Different disciplines and subdisciplines have very different approaches to conferences, from the prestige associated with conference presentation to the process by which one decides what papers are to be presented at the conference. In some disciplines, a conference is a place to test out ideas, and the submissions are reviewed lightly, perhaps primarily for theme. In contrast, in many subdisciplines of computer science, submissions are reviewed carefully, by multiple reviewers, the acceptance rate is comparatively low [1], and the proceedings are considered an archival form of publication [2]. If I recall correctly, there’s a report from the National Academy of Arts and Sciences that says that computer scientists value conference publications because they allow us to get results out in a much more timely fashion than journal publication permits.

This week, I’ve been playing my part in the review process for one conference, SIGCSE [3]. To explain that part, I need to explain more about the SIGCSE reviewing process. SIGCSE is a relatively competitive conference with about a 30% acceptance rate for papers. Each SIGCSE paper gets five or six reviews [4]. Then an associate program chair (APC) moderates a discussion amongst the reviewers and, after a week or so of discussion, writes a meta-review that summarizes the strengths and weaknesses identified the by the reviewers and makes a recommendation. The Program Chairs then use the meta-reviews to put together the conference program, making adjustments as appropriate [5]. I don’t envy them that job.

Anyway, this year, as in many years [7], I’m serving as an APC for SIGCSE [8]. When I first served as an APC, we had to write meta-reviews for somewhere over twenty papers. It’s gotten better. Now we only have to do eight or nine papers. However, we do have to spend more time moderating and encouraging discussion than we used to.

We’re close to the deadline for meta-reviews, so I spent much of today writing those meta-reviews. In order to write a meta-review, I revisit the paper and the reviews, reflect on the positives and negatives the reviewers have raised, consider further issues raised in discussions, and attempt to put that all together in a comparatively concise piece that serves both the conference chairs and the authors. While my primary job is to reflect the recommendations of the reviewers, I will also insert some of my own comments when I think they will help the authors prepare better final papers for this year or revised submissions for the future [9].

Writing meta-reviews takes time. If I count correctly, I wrote about 2600 words of meta-reviews today [10], or about 325 words per meta-review. I know from experience that a tightly-packed, 9pt, two-column, 8.5x11 inch page [11] has about 850 words and I see from the Interweb that a more traditional single-spaced page of text has about 500 words.

Five pages of writing should be enough for a day. But I also had a musing to write. Hence, you get this musing. I had initially assumed it would be shorter [12], but it appears that my muse wanted me to provide you with more of the backstory.

Let’s hope that I can get five pages of real writing done tomorrow. And let’s hope that after those five pages, I’ll still have some ability and inclination to muse.

[1] I believe it’s as low as 10% (and perhaps lower) for some top-tier conferences. However, my quick check suggests that the acceptance rate for the top conferences in theory is about 25%, the acceptance rate for CHI, the top conference in human-computer interaction is also about 25%.

[2] That means that you don’t get to republish the work elsewhere.

[3] Officially, the ACM Technical Symposium on Computer Science Education, or so I believe.

[4] Once in a while, they get only four reviews.

[5] I considered writing about the adjustments they make. But I’ve never been a SIGCSE program chair [6], so I don’t know that what I’d write would be accurate.

[6] And I’ll probably never be a program chair.

[7] If I count correctly, this is my seventh year as an APC.

[8] Am I allowed to mention that? I think so. SIGCSE usually publishes a list of the APCs. It also appears in my CV.

[9] From my perspective, the review process has two primary goals: It helps the conference select the better papers, and it helps the authors improve their work.

[10] Yes, I also edited what I wrote, at least a bit.

[11] That is, a typical page in the SIGCSE conference proceedings.

[12] I wrote reviews today. They totaled 2600 words. I have no writing energy left to muse. Deal.

Version 1.0 of 2018-09-25.