Frictionless peer review

The continuing proliferation of journals can make reviewing frustrating.  Every time you need to carry out a review for a different journal you need to remember the login details for the reviewing system for that specific journal.  More often than not you forget your login, and have to reset your password all over again each time you carry out a review.  Despite the extra hassle, their is no increased security over and above the security on your email login as the password reset details are sent via email.

We have tried to take as much of this hassle out of peer reviewing at Academic Karma.  In fact its possible to carry out a review at Academic Karma without logging in at all.  If an editor invites you to review a paper, we include a unique link for you to accept or decline the invitation (without log-in) and once you have accepted you will be sent an email which contains the review form itself.  You can simply fill in this form using your email client, and submit the review directly from your email.

If you want to use Academic Karma even though the editor didn’t invite you via Academic Karma, you still can, you just need to forward the review request email to your unique Academic Karma forwarding address and this will auto-generate a review invitation sent from Academic Karma.

Finally, remember that to sign up to Academic Karma you don’t need to create a login/password if you have already signed up to ORCID  – you can simply login with your ORCID credentials.

Tools to help you do less peer review and get less peer review spam

At Academic Karma we don’t think that more peer review is necessarily better.  Yes, peer review is essential for the modern scientific enterprise.   But you only have so much time, and there are  many demands on that time: admin, teaching, supervision on top of actually doing your own research.  You could always review more by spending less time on each review – but this will impact on the quality of your peer review.

So, here are the tools we have devised to help you peer review less (but hopefully better!)

1. Combined reviewing/publishing profile so you and others can see that you have done your fair share.  We integrate your publishing record (via ORCID) with a summary of your peer review history (e.g. http://academickarma.org/0000-0002-6982-4660 ).  So you can see when you have done enough peer-review overall, and also when you have done enough peer review for a particular journal. If you have done your share, then you can feel justified in declining the review invitation.

2. A tool to see if the authors have done their fair share of peer review.   When you register an manuscript review invitation at Academic Karma, we generate a combined reviewing and publishing profile of the first and last two authors.  So you can see if the authors have done their share.  If they haven’t then again you are justified in declining the invitation.

3. A tool to decline the review invitation and help editors find new reviewers.   We provide a link for you to ‘decline’ the invitation.  This sends an email to the editor informing them you can’t do the review as well as a link with tools for them to find a new reviewer.  After all, its not really your job to find reviewers for the editors.  The editor will also be able to access your profile (to see that you have done your share) as well as the combined authors’ profile.

4. Keywords to indicate the topics you care about reviewing. We let you specify keywords on your profile, and we provide tools for editors to find reviewers on the basis of those keywords, so that you only get invitations on topics which are of relevance to your current research interests (and not something you worked in 5 years ago).

We are working towards building a platform which provides reviewers much more control over when and what they are asked to peer-review.   Our vision is for an editor to immediately be able to identify willing expert reviewers so that there is no more need to ‘spam’ review invitations to 5 times the number of reviewers required for a paper.

Tutorial: How to use Academic Karma to review a paper

Lets say you are asked to review a paper ‘Test paper 26-1-22-05’ for the journal ‘Test journal’.  The first step is to enter the paper details on the ‘New Review’ tab at http://academickarma.org.    Alternatively, you can forward the review email invitation to a personalised  academickarma.org email address (which you can find on the ‘MyAccount’ page)  and the relevant details will be parsed from the email.

entry

Once the details have been entered you are given the opportunity to decline or accept the review invitation. Lets say you have several review requests.  Academic Karma helps you to decide which paper to prioritise for review by calculating a ‘Combined profile’ listing all the reviewing and
publishing activity of up to 4 of the authors (by default the first and last two).

accept

When you click on the ‘Combined profile’ link you can see that the reviewers have earnt more karma than they have spent, which might convince you to accept this review assignment over another from authors who have do not have a demonstrable record of peer review.

combined

Once you accept the invitation, you are taken to a review page, which is customised to the journal you are reviewing for.  This form is also sent to your email address so that you don’t have to log back in to complete the review.  When you complete the review you are given the option of ‘signing’ the review.

reviewform

An email is sent to the author informing them that your review is complete (but keeping your identity anonymous unless you opted to ‘sign’ your review).

toauth

This is what the review looks like when the author signs into their account.  Note that only the comments to the author are shared.

forauthor

An email is also sent to the editor and journal editorial office with the contents of the review.

toed

That’s it!

Speeding up anonymous peer review by improving communication between author and reviewer

One of the problems of anonymous peer review is that the feedback loop is so slow.    It is almost as if we are back in the early 19th century, the reviewer is in England, the author is in Australia and all correspondence is sent by boat.

Of course, this is the 21st century, and we do have channels of instant communication but the constraints of anonymous peer review mean that a reviewer can’t just dash of an email to an author to request clarification without compromising their anonymity, nor can an author write to a reviewer as they wouldn’t know where to send it.

We have solved this problem at Academic Karma.  Once a paper is registered at Academic Karma with author and reviewer specified (the paper can be registered by the editor or reviewer), we provide an email address specific to the author, editor and reviewers of that paper.  For example, the reviewer can send an email to ‘author_#id@academickarma.org’  (where #id is a unique numerical identifier assigned to the paper).  The author will then receive the email from ‘reviewer.1_#id@academickarma.org’ and can reply to the request for clarification.  All correspondence is also cc’d to the paper editor.  These are not general purpose email addresses – only emails from the author, reviewer or editor will be forwarded and all others will be bounced back to the sender.

We anticipate that this will be useful both to the reviewer (eg. to ask for clarification, to request more data, to request software to evaluate), as well to the author (e.g. to ask for clarification on revision requests, to provide extra data).  We hope that reviewers and authors will have a constructive discourse via this channel however we are also aware of the risks  of allowing communication between author and reviewer, particularly if the review was unfavourable.   Users sending inappropriate or aggressive email will be warned and potentially blocked from the service.

A currency for peer review: PubCreds and Academic Karma

In 2010, Jeremy Fox and Owen Petchey proposed an innovative idea – fix peer review by introducing a peer review currency, which they called ‘PubCreds‘[1]. Fox and Petchey noted that peer review suffers from a ‘tragedy of the commons’ , in which “individuals have every incentive to exploit the ‘reviewer commons’ by submitting manuscripts, but little or no incentive to contribute reviews. The result is a system increasingly dominated by ‘cheats’ (individuals who submit papers without doing proportionate reviewing), with increasingly random and potentially biased results as more and more manuscripts are rejected without external review.” Their solution was to ‘privatise the commons’ by introducing a currency which is earnt by reviewing and spent by getting reviewed.

Symptoms of the ‘tragedy of commons’ in peer review:

One of the main symptoms is slowing down communication of science. Fox and Petchey describe other symptoms, including an increasing tendency for journals to peer review only a small fraction of papers received, resulting in greater randomness in what eventually gets published. Another symptom is editors inviting many more reviewers than necessary in order to secure the minimum number necessary (anecdotally ~5x as many).

Other solutions to ‘tragedy of commons’

Garrett Hardin, when he identified the ‘tragedy of the commons‘ in 1968 [2], concluded that appeals to altruism do not work, and that the only solution is ‘mutual coercion, mutually agreed upon’. This conclusion seems to fit the status quo in peer review, where most reviewing is done on the basis of friendly editorial coercion.

Its worth commenting very briefly on post-publication peer review. Post-publication peer review addresses one of the main symptoms – slowing down scientific communication – by publishing the manuscript before it is reviewed. However, by itself, this does not address the underlying ‘tragedy’. Open peer review – often a feature of post-publication peer review – does address the tragedy by providing a platform for the reviewer to generate a citable output demonstrating their expertise. However, open peer review also carries a (real or perceived) associated cost of potential animosity from the author, which for many reviewers outweighs the incentive.

PubCreds – a peer review currency

Petchey and Fox suggest solving the ‘tragedy’ by introducing a peer review currency (PubCreds) . This would require academics to accept enough review assignments to support the rate at which they are submitting papers. Petchey and Fox identified lots of potential issues to think through: what about early career researchers who haven’t had the chance to accumulate PubCreds? ; what about multi-author papers?; what about re-reviews?; is an overdraft allowed? ; are account balances publicly viewable?

However the big unresolved question remains how do we establish such a currency? This is not trivial – a currency has no value until you can use it to buy something. Petchey and Fox suggest a top-down approach – i.e. publishers mandate a currency. However, publishers, with 30-40% profit margins, have very little incentive to introduce a peer-review currency which would unlikely yield any financial gain but would incur significant costs.

Academic Karma as an implementation of PubCreds

Academic Karma is essentially a community driven implementation of PubCreds. We call the peer review currency ‘karma’, reflecting the underlying philosophy that if you carry out good quality, timely peer review for others you can expect they will do the same for you. In order to make the implementation work we built from a reviewing platform which integrates with the reviewing procedure at almost any journal. As a reviewer you can submit a review to any journal through the system to earn karma. As an editor, you can invite reviewers and co-ordinate the process using Academic Karma.

Ongoing challenges

The biggest challenge Academic Karma faces is automated peer review systems at the journals. Currently the reviews generated using Academic Karma are transmitted via email. This works fine for journals where the email actually goes to the handling editor (which tends to be the higher impact journals), but this is not how ‘high-throughput’ peer review systems work.  These systems work by minimising the interaction between editor and reviewer (replacing genuine correspondence with automated emails) to the extent that an email from the reviewer to editor can jam the process.

However, we are making progress better integrating Academic Karma better into publishers’ processes. Currently, we  make sure that the review forms reviewers fill in at Academic Karma match the journal review forms; and that all correspondence is copied to the editorial office as well as the handling editor.   To date we have facilitated 39 reviews at Academic Karma for journals from a range of publishers including Gigascience, Biomed Central, PloS and Nature, but we hope to expand this in 2015!

1.Jeremy Fox and Owen L. Petchey 2010. Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons. Bulletin of the Ecological Society of America 91:325–333. http://dx.doi.org/10.1890/0012-9623-91.3.325

2. Hardin, Garrett. “The tragedy of the commons.” science 162.3859 (1968): 1243-1248. http://dx.doi.org/10.1126/science.162.3859.1243

Fixing peer review

Today Academic Karma comes out of beta with the launch of a new website and we thought it was the perfect opportunity to explain what we think is wrong with peer review, why this is bad for science and how Academic Karma aims to fix peer review.

Peer-review is – at its best – a cornerstone of science. Good peer review identifies potential weaknesses in scientific work, encourages authors to do further work to provide convincing evidence if necessary, and helps to ensure that details required for others to understand and replicate experiments are presented. Good peer-review should lead to greater reproducibility and fewer retractions.

While publication is heavily incentivized – and publication rates continue to grow dramatically (e.g. doi:10.1371/journal.pone.0068397 ) – there are meagre rewards for good peer-review. Editors recount stories of prolific scientists who barely peer-review at all. The reviewing academics do is largely on the basis a sense of duty or friendly coercion from editors. As a result peer review is rarely prioritised and leads to long delays in publishing.

As explained in our animation and  faq, Academic Karma is a universal peer review platform. This means that as a reviewer, you could now do all your reviewing, for whatever journal, using Academic Karma. You would then have a complete copy of all your peer review in a single place, and you would have a public profile listing the number of times per calendar year you reviewed for each journal. Unfortunately there is currently one caveat – the journal has to accept your review via email. Academic Karma will customise the review form to match the format required by any journal, which in our experience makes it possible for most journals to accept the review. If the journal is not willing to process a review which contains all the relevant information simply because you sent it via email, then do you really want to review for that journal anyway?

But the point of this blog is not to sell you on why you should use Academic Karma to improve your experience of peer-review (although we think it will!) but how a universal peer review platform like Academic Karma can help fix peer-review.

The first way is that it can change the incentive structure around reviewing. Academic Karma does this using a reviewing currency (which we call karma). When you review you earn karma, and when your manuscript is reviewed you pay karma. By keeping a positive balance you are reviewing enough to support the rate at which you are publishing. As an extra incentive, if you have a positive balance you can see reviews for your paper as soon as they are completed by the reviewer (i.e. without waiting for all of the other reviewers and editors to finish). Karma can then be awarded either on the basis of timeliness and quality of a review. We are currently transferring karma on the basis of reviews completed within 10 days (50 karma) or 20 days (25 karma) to incentivise timeliness, but it would be possible to incorporate quality measures into this transfer.

The second way is that it keeps a publicly visible profile of an academic’s contribution as a reviewer. This provides extra visibility for reviewing activity, which we hope over time will be incorporated into the ways in which academics are evaluated.

The third way is that it decouples co-ordination of peer review from publishing. We sometimes mistakenly believe that we are peer reviewing ‘for’ a particular journal, when in fact we are peer reviewing for our peers, and for the scientific literature and science in general. However, peer review as a process has become something which is tightly tied to a journal. As a result, authors often have to re-start the peer review process with completely new reviewers when they submit a revised manuscript, even when many of the revisions have been done in response to reviewers reviewing ‘for’ the first journal. A universal peer review platform allows authors to keep the same reviewers even when resubmitting to a different journal. More broadly, however, it is worth noting that journals have not had to innovate much in terms of peer review – there has been substantially more innovation in other areas of publishing. By decoupling peer review co-ordination from publishing (while still allowing complete editorial control), it suddenly becomes possible to innovate purely in the area of peer review. This is exactly what Academic Karma is going to have to do if it is going to succeed in building a critical mass of enthusiastic peer-reviewers.

You can now sign in and register at  Academic Karma using your ORCID credentials – so there is no need to remember yet another password/username.  We hope this platform is a big step towards fixing peer review, but at the end of the day it is just a platform and we really need your support to turn it into a movement for reforming peer review.