[RQC-interest] Review Quality Collector (RQC) status update October 2014

["Lutz Prechelt" <prechelt@inf.fu-berlin.de>]

In the previous update [1] you read that I was getting
involved with the people designing and implementing
the Reviewer Recognition Program (RRP) at Elsevier,
expecting that RRP will pick up some, many, or most
of the RQC ideas.

RRP has now taken me on board formally in the role
of advisor and has published an article about
RRP [2] on the "Reviewer's Update" website.

An RRP pilot has now been running for several months,
with about a dozen journals taking part.
For some hundred manuscripts, when the reviews came in,
the journal's editor was asked to grade the quality
of the review on a scale from one star to five stars
(an approach that is _very_ different from the 
 RQC ideas).

Participation has been about 40%.
We talked to some of those editors in a phone conference
and learned the following things:
- The query often came at the wrong time for them
  (should be at decision-making time, not review reception time)
- They basically believed quality grading was a sensible idea.
- When I explained the RQC approach of 
  - separating "helpful for editor" from "helpful for authors"
  - grading review quality based on several facets for each of these
  - expressing each facet in terms of verbally described levels
  - and converting the resulting grades into percentiles
    for avoiding grade inflation and aiding grade comparability
  the editors appeared open for the first three.
  Regarding the fourth (percentiles), one editor remarked he
  could never do this: "If I tell half of my reviewers that
  they are below average, they will never give me another review
  again."  That editor was from a medical journal.

The latter point was an important learning point for me:
RQC will probably need some kind of selective opt-out mechanism
that allows reviewers to avoid embarrassing themselves.

The next thing we plan is to formally survey the
RRP pilot's editors in order to understand the potential
of RQC-style quality grading better.
I will keep you posted.

Please tell me if you know anybody else who should be
on this mailing list [3].

[1] https://lists.fu-berlin.de/pipermail/rqc-interest/2014/msg00003.html 
[2]
http://www.elsevier.com/reviewers/reviewers-update/elseviers-reviewer-recogn
ition-platform-prepares-for-next-phase 
[3] https://lists.fu-berlin.de/listinfo/rqc-interest 

With kind regards,

  Lutz Prechelt

Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de
Institut f. Informatik; Freie Universitaet Berlin
Takustr. 9, R.014; 14195 Berlin; Germany
+49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/