From prechelt@inf.fu-berlin.de Thu Jan 23 10:40:56 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1W6GmN-0005eF-2H>; Thu, 23 Jan 2014 10:40:55 +0100 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1W6GmN-000fBy-0U>; Thu, 23 Jan 2014 10:40:55 +0100 Received: from busan.imp.fu-berlin.de ([160.45.118.24] helo=BUSAN) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with smtps (envelope-from ) id <1W6GmM-001V1P-Uf>; Thu, 23 Jan 2014 10:40:54 +0100 From: "Lutz Prechelt" To: Date: Thu, 23 Jan 2014 10:40:56 +0100 Message-ID: <008801cf181f$373a63a0$a5af2ae0$@fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac8YHzcePnNdmIU3Qaakg/8JksiN/Q== Content-Language: de X-Originating-IP: 160.45.118.24 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1390470055-0000CD40-4AD1DCE6/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.384998, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED,LOTS_OF_MONEY X-Spam-Checker-Version: SpamAssassin 3.3.3-zedat0a54d5a on Benin.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Community funding approach for Review Quality Collector (RQC) X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2014 09:40:56 -0000 Dear RQC contacts, What you already know ====================== http://www.inf.fu-berlin.de/w/SE/ReviewQualityCollectorHome - Discussing the Review Quality Collector idea with you, I found a lot of interest and received support letters from 6 of the largest scientific publishers (together representing more than 7300 journals) and from 9 research institutions and from individual researchers of another 25 research institutions from 14 countries - I submitted a proposal for public funding (in a very suitable program), but it was rejected - I told you I would develop an idea for community funding and contact you again (which I do today). New: The community funding approach =================================== - With community funding (from publishers and universities), the project can focus more on technical issues and less on adoption and so can be compressed from 3 to 2 years. - The required budget then is 456 000 EUR. (The project will not start if this budget is not achieved by November 2014. If it starts, RQC will be developed and run by a not-for-profit company.) - To this budget, each participating publisher will contribute at most EUR 200 per journal (for all of its journals) and each participating university will contribute at most EUR 70 per post-doctoral researcher (for all of those). - Why are these amounts "at most"? If the sum of these contributions grows beyond 115% of the budget (EUR 524 000), the above contributions are reduced in proportion. - Example: If publishers with 4000 journals and universities with 2000 researchers participate in the funding, the contributions would amount to 4000*200 + 2000*70 = 940 000 EUR so the contributions will be adjusted by factor 524000/940000 = 0.557 and so become EUR 111 per journal and EUR 39 per researcher. - Benefits for the participating funding partners: - can co-control the design of the RQC service - have priority usage rights (everybody else can start not before year 3) - reap the marketing value of being an innovator (RQC will forever advertise them as founding sponsors) - receive 60% fee reduction during 5 years of subsequent use (probably worth about half of the funding in total) Questions are extremely welcome. I have three questions as well: - Under which circumstances do you think would your company or university participate in this funding scheme? - What would be its primary motivation to do so? - What might be the primary inhibitor against doing so? Please speak up! Reply email will go to me, not the mailing list. I will be at the Academic Publishing Europe conference next week and will talk to some of you in person there. Hoping to hear from many of you, Lutz Prechelt Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de Institut f. Informatik; Freie Universitaet Berlin Takustr. 9, R.014; 14195 Berlin; Germany +49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/ From prechelt@inf.fu-berlin.de Tue Mar 25 15:08:19 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1WSS1Z-002spC-NB>; Tue, 25 Mar 2014 15:08:17 +0100 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1WSS1Z-001aKP-LO>; Tue, 25 Mar 2014 15:08:17 +0100 Received: from busan.imp.fu-berlin.de ([160.45.118.24] helo=BUSAN) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtpsa (envelope-from ) id <1WSS1Z-004C2Y-JK>; Tue, 25 Mar 2014 15:08:17 +0100 From: "Lutz Prechelt" To: Date: Tue, 25 Mar 2014 15:08:18 +0100 Message-ID: <011d01cf4833$abec8450$03c58cf0$@fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac9IM6vaWxpgiLkVSAeZ1UrM4wqrGg== Content-Language: de X-Originating-IP: 160.45.118.24 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1395756497-0000CD40-9D1B3221/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.017875, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED X-Spam-Checker-Version: SpamAssassin 3.4.0 on Gabun.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Review Quality Collector (RQC) initiative: Status X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 14:08:19 -0000 Hi all, last thing you heard (in January) was that the public funding proposal failed and that I was going to talk to some publishers at the Academic Publishing Europe conference about possible community funding https://lists.fu-berlin.de/pipermail/rqc-interest/2014/msg00000.html Well, I did. - The reactions to RQC were all positive. Everybody felt that RQC is something that ought to exist. The willingness to fund the RQC creation project, however, was not so high. - I was pointed to Fred Dylla and the STM association as an organization that might be willing to fund it instead or help find other solutions. - I wrote a short proposal to him, but the required project volume was far larger than what they would usually do. - In the meantime, Elsevier approached me and asked whether I would join their own reviewer incentives initiative. This grew out of the winning entry of their 2012 Peer Review Challenge contest http://www.peerreviewfuture.com/?page_id=242 It is currently oriented towards quantifying only the amount of reviewing, not the quality, but is interested in becoming quality-oriented as well, possibly picking up some of the RQC ideas. This is under discussion. - Fred Dylla also suggested to perhaps apply for funding from the EU for a consortium project. I will investigate this possibility in the coming weeks. If you (publisher or research institution) are or might be interested in joining such a consortium, please send me a note! That's it for today. Questions and feedback are highly appreciated. With kind regards, Lutz Prechelt Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de Institut f. Informatik; Freie Universitaet Berlin Takustr. 9, R.014; 14195 Berlin; Germany +49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/ From prechelt@inf.fu-berlin.de Tue Apr 22 14:16:17 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1WcZcU-003F2R-2u>; Tue, 22 Apr 2014 14:16:14 +0200 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1WcZcU-002Ehz-18>; Tue, 22 Apr 2014 14:16:14 +0200 Received: from busan.imp.fu-berlin.de ([160.45.118.24] helo=BUSAN) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtpsa (envelope-from ) id <1WcZcT-001T9v-V2>; Tue, 22 Apr 2014 14:16:14 +0200 From: "Lutz Prechelt" To: Date: Tue, 22 Apr 2014 14:16:15 +0200 Message-ID: <00c501cf5e24$a879ec70$f96dc550$@fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac9eJKhPc2u7fxm6TXC88RaDhKhVsw== Content-Language: de X-Originating-IP: 160.45.118.24 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1398168974-0000CD40-02129D5B/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.004573, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED X-Spam-Checker-Version: SpamAssassin 3.4.0 on Gabun.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Review Quality Collector (RQC) status update April 2014 X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Apr 2014 12:16:17 -0000 In the last update (March 25), you read that I wanted to look for possible EU project calls related to RQC. When I did, I found one promptly: "e-Infrastructure for Open Access" https://ec.europa.eu/programmes/horizon2020/en/news/e-infrastructures-0 The call mentioned "new forms of peer review" as one of the work areas, so RQC ought to fit in as one element of a larger project. Unfortunately, the closing date of that call was April 15, so I started a frantic search for a consortium applying for that call. With the help of some of you I indeed found such a consortium. However, although they showed a somewhat positive attitude towards the RQC idea, it was simply too late to fit it in. We'll keep in contact, though. As for the Elsevier initiative: They are going to visit me in Berlin at the end of May to talk about it in more detail and get to know each other personally. In the coming weeks, I'll also be looking for further EU funding opportunities in other directions (biomedical realm) and possibly also talk to the German science funding agency DFG. That's it for today. As usual, questions and feedback are highly appreciated. With kind regards, Lutz Prechelt Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de Institut f. Informatik; Freie Universitaet Berlin Takustr. 9, R.014; 14195 Berlin; Germany +49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/ From prechelt@inf.fu-berlin.de Mon Jul 14 16:26:01 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1X6hCZ-00323I-NU>; Mon, 14 Jul 2014 16:25:59 +0200 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1X6hCZ-00274Q-MX>; Mon, 14 Jul 2014 16:25:59 +0200 Received: from busan.imp.fu-berlin.de ([160.45.118.24] helo=BUSAN) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtpsa (envelope-from ) id <1X6hCZ-003WDD-LC>; Mon, 14 Jul 2014 16:25:59 +0200 From: "Lutz Prechelt" To: Date: Mon, 14 Jul 2014 16:26:00 +0200 Message-ID: <014601cf9f6f$88e06fb0$9aa14f10$@fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac+fb4jFTvdRhLmzT8ShhqfCGXrCyQ== Content-Language: de X-Originating-IP: 160.45.118.24 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1405347959-00000941-16ABCB4E/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001510, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED X-Spam-Checker-Version: SpamAssassin 3.4.0 on Burundi.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Review Quality Collector (RQC) status update July 2014 X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jul 2014 14:26:01 -0000 In the last update you read that I was slightly too late to get into the consortium for a EU project call that fit well with RQCs goals. I wanted to look for other funding opportunities next. The second stream of activity was getting acquainted with people from Elsevier starting a related initiative. Well, the search for alternative funding sources has not come up with much, so I was starting to be less optimistic about finding funding for the RQC project myself. The meeting with the Elsevier people, however, was positive: In 2012, Elsevier ran a contest, the Peer Review Challenge [1] to collect ideas for improving the peer review system. The first-ranked entry [2] was Simon Gosling's "Reviewer Badges and Rewards scheme" [3] which proposes to publicly recognize reviewers for the quantity of their reviewing. To implement this idea, Elsevier has recently started a "Reviewer Recognition Program" (RRP). We have now agreed that I will work with them to help extend the RRP to also pick up quality evaluation ideas from RQC. Elsevier is initially implementing the RRP in a pilot form with only a few journals and intends to then roll it out more broadly: first across the Elsevier universe and then across the whole publishing industry (as a free service). I will keep you posted. [1] http://www.peerreviewfuture.com/?page_id=14 [2] http://www.elsevier.com/connect/winners-announced-for-peer-review-challenge [3] http://www.elsevierblogs.com/yourpeerreviewfuture/?page_id=242 Lutz Prechelt Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de Institut f. Informatik; Freie Universitaet Berlin Takustr. 9, R.014; 14195 Berlin; Germany +49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/ From prechelt@inf.fu-berlin.de Wed Oct 29 11:08:30 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1XjQAy-002mkQ-BO>; Wed, 29 Oct 2014 11:08:24 +0100 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1XjQAy-0042Ho-AM>; Wed, 29 Oct 2014 11:08:24 +0100 Received: from kaduna.imp.fu-berlin.de ([160.45.118.73] helo=KADUNA) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtpsa (envelope-from ) id <1XjQAy-001U3N-99>; Wed, 29 Oct 2014 11:08:24 +0100 From: "Lutz Prechelt" To: Date: Wed, 29 Oct 2014 11:08:24 +0100 Message-ID: <031f01cff360$46882050$d39860f0$@inf.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: Ac/zWrmvmy3ynJYPQ9KuHWi16KZzjw== Content-Language: de X-Originating-IP: 160.45.118.73 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1414577304-00067E64-226533B6/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000242, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED X-Spam-Checker-Version: SpamAssassin 3.4.0 on Gabun.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Review Quality Collector (RQC) status update October 2014 X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Oct 2014 10:08:30 -0000 In the previous update [1] you read that I was getting involved with the people designing and implementing the Reviewer Recognition Program (RRP) at Elsevier, expecting that RRP will pick up some, many, or most of the RQC ideas. RRP has now taken me on board formally in the role of advisor and has published an article about RRP [2] on the "Reviewer's Update" website. An RRP pilot has now been running for several months, with about a dozen journals taking part. For some hundred manuscripts, when the reviews came in, the journal's editor was asked to grade the quality of the review on a scale from one star to five stars (an approach that is _very_ different from the RQC ideas). Participation has been about 40%. We talked to some of those editors in a phone conference and learned the following things: - The query often came at the wrong time for them (should be at decision-making time, not review reception time) - They basically believed quality grading was a sensible idea. - When I explained the RQC approach of - separating "helpful for editor" from "helpful for authors" - grading review quality based on several facets for each of these - expressing each facet in terms of verbally described levels - and converting the resulting grades into percentiles for avoiding grade inflation and aiding grade comparability the editors appeared open for the first three. Regarding the fourth (percentiles), one editor remarked he could never do this: "If I tell half of my reviewers that they are below average, they will never give me another review again." That editor was from a medical journal. The latter point was an important learning point for me: RQC will probably need some kind of selective opt-out mechanism that allows reviewers to avoid embarrassing themselves. The next thing we plan is to formally survey the RRP pilot's editors in order to understand the potential of RQC-style quality grading better. I will keep you posted. Please tell me if you know anybody else who should be on this mailing list [3]. [1] https://lists.fu-berlin.de/pipermail/rqc-interest/2014/msg00003.html [2] http://www.elsevier.com/reviewers/reviewers-update/elseviers-reviewer-recogn ition-platform-prepares-for-next-phase [3] https://lists.fu-berlin.de/listinfo/rqc-interest With kind regards, Lutz Prechelt Prof. Dr. Lutz Prechelt; prechelt@inf.fu-berlin.de Institut f. Informatik; Freie Universitaet Berlin Takustr. 9, R.014; 14195 Berlin; Germany +49 30 838 75115; http://www.inf.fu-berlin.de/w/SE/ From prechelt@inf.fu-berlin.de Wed Nov 19 14:09:12 2014 Received: from outpost1.zedat.fu-berlin.de ([130.133.4.66]) by list1.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1Xr50Q-0024Q0-4C>; Wed, 19 Nov 2014 14:09:10 +0100 Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtp (envelope-from ) id <1Xr50Q-000MDb-3C>; Wed, 19 Nov 2014 14:09:10 +0100 Received: from kaduna.imp.fu-berlin.de ([160.45.118.73] helo=KADUNA) by inpost2.zedat.fu-berlin.de (Exim 4.82) for rqc-interest@lists.fu-berlin.de with esmtpsa (envelope-from ) id <1Xr50Q-0022mj-1t>; Wed, 19 Nov 2014 14:09:10 +0100 From: "Lutz Prechelt" To: Date: Wed, 19 Nov 2014 14:09:10 +0100 Message-ID: <018001d003fa$01cf8930$056e9b90$@inf.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AdAD+YnPhvki8C0iSzO0i42ixWSGrw== Content-Language: de X-Originating-IP: 160.45.118.73 X-purgate: clean X-purgate-type: clean X-purgate-ID: 151147::1416402550-00067E64-EB0F9742/0/0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.220283, version=1.2.4 X-Spam-Flag: NO X-Spam-Status: No, score=-50.0 required=5.0 tests=ALL_TRUSTED X-Spam-Checker-Version: SpamAssassin 3.4.0 on Algerien.ZEDAT.FU-Berlin.DE X-Spam-Level: Subject: [RQC-interest] Review Quality Collector (RQC) status update November 2014 X-BeenThere: rqc-interest@lists.fu-berlin.de X-Mailman-Version: 2.1.16 Precedence: list List-Id: sporadic information on Review Quality Collector List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2014 13:09:12 -0000 In the previous update [1], I told you that we were going to run a survey regarding the participating editor's perceptions of the first Elsevier review quality grading pilot experiment (and of their review quality situation in general). This will be done in December. We are also looking for participants now for a second round of review quality grading pilot use, where we hope to move from the current simple grading mode a bit towards the more refined ideas of Review Quality Collector [2], although that will also depend on the survey outcomes. So we are looking for editors who - are interested in discussing review quality issues with us, - would like to try out review quality grading and feedback in their journal, and - are editing an Elsevier journal (in whatever area). * Are you such a person? Then please speak up! * Do you know such a person? Or do you know who might know such a person? Then please forward this email to him or her. And also, as usual: Please tell me if you know anybody else who may like to be on this mailing list [3]. Lutz [1] https://lists.fu-berlin.de/pipermail/rqc-interest/2014/msg00004.html [2] http://www.inf.fu-berlin.de/w/SE/ReviewQualityCollectorHome [3] https://lists.fu-berlin.de/listinfo/rqc-interest (you can subscribe here)