IJCAI-ECAI 2018: FAQ about notifications


REVIEW AND DECISION PROCESS


1. How many reviews should I receive?

In most papers cases: four, among which one by a senior program member committee (SPC), which is sometimes (but not always) short and focusing on high-level details. Sometimes, only three. Sometimes, additional reviews were added (before or after rebuttal). In addition to these, you will sometimes receive a metareview (see further).

2. How was the decision taken?

Except when the decision was pretty obvious, there was a discussion among reviewers about your paper. The senior program member committee was in charge of leading it. After the discussion was completed, the SPC made a recommendation reflecting the outcome of the discussion (or, in rare cases when the decision was inconclusive, they expressed their impossibility to make a recommendation). Then the decisions was passed to an area chair, who supervised 50 to 100 papers and whose role was to make the decision process consistent and homogeneous through different SPCs and different areas. In some cases, the paper, together with its reviews and discussions, was seen by several area chairs and/or to the program chair.

3. What is the role of the metareview?

The metareview is optional. It summarizes the discussion and/or explain the decision on your paper.  If your paper is accepted conditionally on minor revisions, it should also explain what has to be added or corrected in the final version. If you cannot find a metareview, it probably means that there was no significant information to add to that already contained in the reviews.

Sometimes you will find not only one, but two metareviews (one from the SPC and one from an area chair). Normally, the two metareviews are consistent, but in some rare cases, they might appear conflicting: in that case, the metareview provided by the area chair is more recent and prevails.

4. The scores of the reviews on my paper are high, and yet my paper has been rejected. Why?

Scores aren't everything, and there is a huge disparity of scores across reviewers. Some reviewers give very high scores like 8 or 9 with reviews that are only mildly positive, while some others give enthusiastic reviews with several good reasons to accept, with a score of 6. The qualitative information contained in the reviews, the output of the discussion, and the opinion of one or several area chair(s) when necessary, counted at least as much as the scores.

I got several questions of the form (I'm just copying one but I got at least 5 of these): 

" Our paper got 9, 8, 8, and 5 from four different reviewers. it has been rejected. I know two other papers were accepted with scores 7,5,5 and 5 (average: 5.5) and 7,7,8,5 (average:6.75)."

If the decisions were based on the average score, this would be simple and would save us a lot of time reading reviews and discussing papers.
The decisions are not based on scores, but on the qualitative appreciation of the comments contained in the reviews. The senior PC members and area chairs have carefully read the reviews and made decisions according to these comments. Scores are sometimes useful, but they don't count as much as you think. There are extremely variable across reviewers: some reviewers give a 8 or a 9 while writing a review with few positive comments, while I have seen truly positive reviews (with a few minor criticisms) with a global score of 5. Scores are helpful to give a first feeling about whether the reviews are rather positive or rather negative, but that's all. Some paper were rejected with an average score as high as 8 while some other papers were accepted with an average score below 5.

Sometimes you may find a "0" score associated with a metareview (or sometimes a review). This means that the metareview comes form an area chair who did not give any score. This zero was not taken into account for computing average scores!


5.  Still, the reviews are globally positive, and yet my paper has been rejected. Why?

IJCAI-ECAI 2018 is an top-level archival conference. You know what top-level means, and archival means that the paper must have (almost) the quality of a journal paper. If your paper developed a nice idea and has good results but the exposition of the work is not clear, or it is insufficiently positioned with respect to related work,  or there are flaws that prevent the paper to published in this state, then it is likely that you received positive and/or encouraging reviews but that your paper has not been accepted. In the same way as a journal paper is rarely accepted at the first attempt, an IJCAI (or AAAI) paper may have to be submitted two or three times before being accepted. You may resubmit it very soon to another conference. If that tradition continues, it is possible that AAAI-19 makes it possible to attach the IJCAI-18 reviews and how you responded to it. I have seen lmany examples of papers with attached AAAI-18 reviews that were accepted to IJCAI-ECAI 2018 after the reviewers noticed that the AAAI reviews had been taken into account adequately.

I got a few comments of this type:

"Three reviewers out of four wanted to accept my paper and only one voted for rejection. Therefore it should have been accepted. "

Selecting papers is not a voting process. We accept a paper once it is close to be perfect enough so that it can be stored in archival conference proceedings. If one reviewer complains abut important issues such as unclear results, insufficiently described experiments, or a missing comparison with related work, then you must first revise your paper before it is accepted.

Accepting a paper to a conference has some similarities with accepting a paper to a journal. If you submit a paper to a journal, and three reviewers out of four think it should be accepted but the last one has come justified complains, then the paper will not be accepted but will have to undergo revisions. It will be eventually accepted once all reviewers are happy with the way the authors solved all the important concerns they had. You may consider the sequence of IJCAI and AAAI conference as a journal, with the difference that the reviewers are not necessarily the same from one submission to the next one. But I can tell you that when a paper is good, is generally manages to get accepted after, say, two or three attempts. If you had submitted to a journal, given the longer delays, you would have had to wait the same amount of time before it is eventually accepted. Getting sometimes a rejection that you view as unfair, and that sometimes is, may simply be the price to pay for having a quick review process. 



COMPLAINING

Every program chair has a long career behind him and had seen many of their papers rejected, sometimes in a very unfair way. I very much undertstand what you may feel if your paper has been rejected: keep in mind that I have felt it many times, sometimes before you were born (when notifications were printed and sent by post!).  It is normal that your first reaction is to complain. It is possible to do so. Except in very speciifc cases explained below, it cannot change the decision, but it can help me seeing what is wrong with the review process, or sometimes spot reviewers who should perhaps not be asked again next time.

So: yes, it is possible to complain. However: out of 3470 submitted papers, 2750 have been rejected. You understand that if I receive more than , say, 50 messages, I won't be able to answer them. Please write me only when it is really, really deserves it. Details are below.

6.  There was a procedural error in handling my paper. What should I do?

A procedural error (like in court) is something that does not relate to scientific issues, but to a dysfunctioning of the review process, such as:

- your paper was rejected because of the existence of a previous ArXiv version or a previous presentation at a workshop, while this was explicitly allowed.

- your paper was rejected because of self-plagiarism, concerning only a small part of the text

- one of the reviews that you received is obviously not a review on your paper, but on another paper

The presence of a procedural error is the only case where we may revert the decision on your paper. If you think that such is the case, please write me an email, whose subject contains your paper id and the keyword PROCEDURAL. You will get an answer in any case.


8.  I really find (some of) the reviews unfair. What should I do?

With 3470 submissions and more than program committee members, the review and decision process cannot be perfect. certainly, some great papers have been rejected. A paper can sometimes be rejected for a reason that you find objectively wrong.

If this reason concerns the general appreciation of the novelty, significance or clarity of exposition of your work, don't write me, I will not answer. For instance, writing me because a reviewer wrote "this algorithm has been used already in similar domains", "the evaluation is not  conclusive", "the impact will be limited", "the related work section should be more extensive", or "there should have been more experiments", and you disagree with this, is useless.

If this reason concerns something that you find unacceptable in a review, such as:

- the initial review mentioned a technical flaw; you have responded to it in the rebuttal, but the final review still mentions this flaw. (This may be useful to clarify this in case you want to resubmit at another conference and append the IJCAI reviews.)

- the review is harsh, impolite or aggressive

- the review is negative while not giving any reason for that

then you may write to me, with the id of your paper in the subject and the keyword OTHER. I will try to answer (but the delay may be long).

There's a case which, unfortunately, it quite frequent: short, uninformative reviews. These reviews have always been neglected in the decision process. It is useless to write about that. It is also needless to write about the zero scores (cf. FAQ 4).


9.  Can I ask the decision to be reconsidered?

If there is a procedural error, yes. In other cases, no. It is useless to ask the decision to be reconsiderd because the reviewers liked your paper less than you. Doing so would need finding new reviewers and start a new review cycle which would take at least two months: this will be done at the next suitable conference.


10.  The reviews contained specific questions that we answered at rebuttal time, but the reviewers did not update their reviews, which suggest that they have not read the rebuttal. What can we do?

You can complain. I will ask the reviewers to update their reviews. Note that in most cases reviewers do read and comment the rebuttal; but then they forget to update their reviews.