IJCAI-ECAI 2018: FAQ about
REVIEW AND DECISION PROCESS
How many reviews should I receive?
papers cases: four, among which one by a senior program
which is sometimes (but not always) short and focusing
on high-level details. Sometimes, only three. Sometimes,
additional reviews were added (before or after
rebuttal). In addition to these, you will sometimes
receive a metareview (see further).
2. How was
the decision taken?
Except when the decision was pretty obvious, there was a
discussion among reviewers about your paper. The senior
program member committee was in charge of
leading it. After the discussion was
completed, the SPC made a recommendation
reflecting the outcome of the discussion
(or, in rare cases when the decision was
inconclusive, they expressed their
impossibility to make a recommendation).
Then the decisions was passed to an area
chair, who supervised 50 to 100 papers and
whose role was to make the decision process
consistent and homogeneous through different
SPCs and different areas. In some cases, the
paper, together with its reviews and
discussions, was seen by several area chairs
and/or to the program chair.
3. What is the role of the metareview?
metareview is optional. It summarizes the discussion
and/or explain the decision on your paper. If your
paper is accepted conditionally on minor revisions, it
should also explain what has to be added or corrected in
the final version. If you cannot find a metareview, it
probably means that there was no significant information
to add to that already contained in the reviews.
Sometimes you will find not only one, but
two metareviews (one from the SPC and one
from an area chair). Normally, the two
metareviews are consistent, but in some rare
cases, they might appear conflicting: in
that case, the metareview
provided by the area
chair is more recent and prevails.
4. The scores of the reviews on my paper are
high, and yet
my paper has
Scores aren't everything, and there is a huge disparity of scores
across reviewers. Some reviewers give very high scores like 8 or 9
with reviews that are only mildly positive, while some others give
enthusiastic reviews with several good reasons to accept, with a
score of 6. The qualitative information contained in the reviews,
the output of the discussion, and the opinion of one or several area
chair(s) when necessary, counted at least as much as the scores.
I got several questions of the form (I'm just copying one but I got
at least 5 of these):
Our paper got 9, 8,
8, and 5 from four different reviewers.
it has been rejected. I know two other papers were accepted
with scores 7,5,5 and 5 (average: 5.5) and 7,7,8,5
If the decisions were based on the average score, this would be
simple and would save us a lot of time reading reviews and
The decisions are not based on scores,
but on the qualitative appreciation of the comments contained in
the reviews. The senior PC members and area chairs have
carefully read the reviews and made decisions according to these
comments. Scores are sometimes useful, but they don't count as
much as you think. There are extremely variable across reviewers:
some reviewers give a 8 or a 9 while writing a review with few
positive comments, while I have seen truly positive reviews (with
a few minor criticisms) with a global score of 5. Scores
are helpful to give a first feeling about whether the reviews are
rather positive or rather negative, but that's all. Some paper were
rejected with an average score as high as 8 while some other papers
were accepted with an average score below 5.
Sometimes you may find a "0" score associated with a metareview (or
sometimes a review). This means that the metareview comes form an
area chair who did not give any score. This zero was not taken into
account for computing average scores!
5. Still, the reviews are globally
yet my paper has
IJCAI-ECAI 2018 is an top-level archival conference. You know what
top-level means, and archival means that the paper must have
(almost) the quality of a journal paper. If your paper developed a
nice idea and has good results but the exposition of the work is not
clear, or it is insufficiently positioned with respect to related
work, or there are flaws that prevent the paper to published
in this state, then it is likely that you received positive and/or
encouraging reviews but that your paper has not been accepted. In
the same way as a journal paper is rarely accepted at the first
attempt, an IJCAI (or AAAI) paper may have to be submitted two or
three times before being accepted. You may resubmit it very soon to
another conference. If that tradition continues, it is possible that
AAAI-19 makes it possible to attach the IJCAI-18 reviews and how you
responded to it. I have seen lmany examples of papers with attached
AAAI-18 reviews that were accepted to IJCAI-ECAI 2018 after the
reviewers noticed that the AAAI reviews had been taken into account
I got a few comments of this type:
"Three reviewers out of four wanted to accept my paper and only
one voted for rejection. Therefore it should have been accepted.
Selecting papers is not a voting process. We accept a paper once it
is close to be perfect enough so that it can be stored in archival
conference proceedings. If one reviewer complains abut important
issues such as unclear results, insufficiently described
experiments, or a missing comparison with related work, then you
must first revise your paper before it is accepted.
Accepting a paper to a conference has some similarities with
accepting a paper to a journal. If you submit a paper to a journal,
and three reviewers out of four think it should be accepted but the
last one has come justified complains, then the paper will not be
accepted but will have to undergo revisions. It will be eventually
accepted once all reviewers are happy with the way the authors
solved all the important concerns they had. You may consider the
sequence of IJCAI and AAAI conference as a journal, with the
difference that the reviewers are not necessarily the same from one
submission to the next one. But I can tell you that when a paper is
good, is generally manages to get accepted after, say, two or three
attempts. If you had submitted to a journal, given the longer
delays, you would have had to wait the same amount of time before it
is eventually accepted. Getting sometimes a rejection that you
view as unfair, and that sometimes is, may simply be the price to
pay for having a quick review process.
Every program chair has a long career behind him and had seen many
of their papers rejected, sometimes in a very unfair way. I very
much undertstand what you may feel if your paper has been rejected:
keep in mind that I have felt it many times, sometimes before you
were born (when notifications were printed and sent by post!).
It is normal that your first reaction is to complain. It is possible
to do so. Except in very speciifc cases explained below, it cannot
change the decision, but it can help me seeing what is wrong with
the review process, or sometimes spot reviewers who should perhaps
not be asked again next time.
So: yes, it is possible to complain. However: out of 3470 submitted
papers, 2750 have been rejected. You understand that if I receive
more than , say, 50 messages, I won't be able to answer them. Please
write me only when it is really, really deserves it. Details are
6. There was a procedural error in
should I do?
procedural error (like in court) is something that does not relate
to scientific issues, but to a dysfunctioning of the review process,
- your paper was rejected because of the existence of a previous
ArXiv version or a previous presentation at a workshop, while this
was explicitly allowed.
- your paper was rejected because of self-plagiarism, concerning
only a small part of the text
- one of the reviews that you received is obviously not a review on
your paper, but on another paper
The presence of a procedural error is the only case where we may
revert the decision on your paper. If you think that such is the
case, please write me an email, whose subject contains your paper id
and the keyword PROCEDURAL. You will get an answer in any case.
8. I really find (some of) the reviews
should I do?
3470 submissions and more than program committee members, the review
and decision process cannot be perfect. certainly, some great papers
have been rejected. A paper can sometimes be rejected for a reason
that you find objectively wrong.
If this reason concerns the general appreciation of the novelty,
significance or clarity of exposition of your work, don't write me,
I will not answer. For instance, writing me because a reviewer wrote
"this algorithm has been used already in similar domains", "the
evaluation is not conclusive", "the impact will be limited",
"the related work section should be more extensive", or "there
should have been more experiments", and you disagree with this, is
If this reason concerns something that you find unacceptable in a
review, such as:
- the initial review mentioned a technical flaw; you have responded
to it in the rebuttal, but the final review still mentions this
flaw. (This may be useful to clarify this in case you want to
resubmit at another conference and append the IJCAI reviews.)
- the review is harsh, impolite or aggressive
- the review is negative while not giving any reason for that
then you may write to me, with the id of your paper in the subject
and the keyword OTHER. I will try to answer (but the delay may be
There's a case which, unfortunately, it quite frequent: short,
uninformative reviews. These reviews have always been neglected in
the decision process. It is useless to write about that. It is also
needless to write about the zero scores (cf. FAQ 4).
9. Can I ask the decision to be
there is a
error, yes. In other cases, no. It
is useless to
decision to be
Doing so would
and start a
would take at
will be done
at the next
10. The reviews contained specific
we answered at
that they have
not read the
can we do?
You can complain. I will ask the reviewers
Note that in