Peer Pre-review

ImageSocial Science One grantees will have the opportunity to participate in a new peer pre-review feedback service offered by Social Science One. Peer pre-review is designed to solve a major problem in science, and science journalism, by speeding up the process of scientific discovery and publication, while reducing the time that papers with flaws are on the web prior to peer review and publication. It offers scholars instant feedback on their draft papers from leading scholars.

To accomplish these goals, Social Science One is building both a community of independent, private peer pre-reviewers (incentivized through funding to participate), as well as partnerships with journal editors to help speed up peer review at scientific journals. Peer pre-review formalizes some of what happens informally among colleagues and friends, with reviews offered to grantees that are fast, complete, reliable, repeatable, and expert. Peer pre-review may be of particular help to scholars at smaller institutions or at universities or regions without many strong scholars in the same field.

The Problem

After researchers draft academic papers, often years before publication in a scholarly journal, they post them on their homepage or in a working paper repository. Reporters then read these drafts and publish news articles about them.  Unfortunately, the probability of a news article appearing depends mostly on how sensational the claims are in the scholarly paper, not only on the veracity of these claims. Incorrect academic paper drafts may then mislead the public, but they can also mislead other academics as the papers stay out there in unpublished and often incorrect form, thus sending the scholarly community down unfruitful paths.

Trying to “fix” journalism will not solve these problems, as journalists are not the problem.  Journalists already work hard trying to weed out papers with mistakes and or ones which other academics they interview think are flawed. Unfortunately, figuring out whether a claim is correct is an extremely difficult scholarly task. We know that one person following all the rules of science cannot accomplish the task on their own, and journalists and their editors are not trained to do it. Indeed, the only solution ever devised that is at all reliable is the scientific community (which works together in competition and cooperation in pursuit of the same goals, and with an incentive structure that apportions credit more for moving the scholarly consensus further from the status quo, and proportionately increasing the evidence necessary for doing so). The whole process recognizes that it is easier to fool yourself than it is to fool someone else, and so the community is required to evaluate claims.  That’s true for every scientist, and every journalist, no matter how smart, diligent, or trained. It requires many people working on the problem together, checking and building on each other’s work.

The result of this process is that a scholarly paper that makes it all the way through the peer review process, is approved by the editor, and gets published in a scholarly journal is perhaps only slightly more likely to be correct than serious unpublished papers. Yet, although the advantage to individual articles may be slight, the aggregate result of repeated application of the process has produced the majority of the technological and scientific knowledge the world has known for the last 400 years. Thus, instead of trying to teach journalists the wrong lesson by asking them to try to individually replace the entire scientific community, we will speed up the process of scientific discovery, in particular the time between the appearance of a draft of a paper and a scholarly publication.   

Let’s first characterize this process, since the proposed solution will also solve a problem for academics as well as for journalists. 

The Process

In the social sciences, when researchers finish paper drafts, they typically make them available to a few colleagues or trusted friends, get comments, revise, and then post it on their websites. Sometimes they also give it at a conference (the hurdle for which is just the title and an abstract, and so not the same as a scholarly publication), and they revise some more on the basis of comments received. They then send it to the journal for peer review. This first round of the peer review process almost always takes at least 3 months, and can take as long as 6 months or more.  Few papers are accepted on the first round, and so there’s some more revision, and then there is (if given a “revise and resubmit”) resubmission to the original journal or (if rejected) a new journal. Then comes another 3-6 month waiting period. It is common for papers from the best scholars to go through multiple rounds or review, during most of which scholars spend their time waiting, and readers spend their time with a paper that may be misleading everyone out there on the web.

For academics, the peer review process is humbling and frustrating. It unquestionably makes papers better, at least on average. It also takes what seems like forever. Authors often think they are misunderstood by reviewers but, if the reviewer missed a crucial point in a paper, the fault is the author’s for not putting their point in a place so a semi-attentive reviewer (like a semi-attentive journal reader) wouldn’t miss it. This whole process is one of the most difficult parts of being a scientist, but it is also one of the most important.

The Solution

Here’s our plan for how we might speed academic publication, thereby reducing the time that papers with flaws are on the web prior to publication.

  1. As a service for all researchers receiving grants, Social Science One offers a voluntary peer pre-review process. Anonymous academics are asked to write serious peer reviews of drafts of their papers within about a week, incentivized with proper compensation.

    1. Pre-reviewers would be asked to write the same type of review they would for a scholarly journal, only much faster, and more complete and constructive.

    2. It rarely takes more than a day to prepare a peer review once underway. The longer wait is merely the time it takes for the journal to find the reviewer and for the reviewers to make time in their schedules to do the work. Social Science One's financial incentive aims to ensure a reviewer accepts the assignment and commits to a  timely turnaround.

    3. Researchers will judge the quality of their reviews, and poorly rated reviewers will not be asked back.

    4. The rate of pay will evolve as Social Science One experiments with what is needed to make this process effective.

  2. Research grantees receive up to three reviewers for each paper, but not simultaneously as with a journal. Instead, it will be sequential, so the paper evolves with each set of comments. (We will also experiment with having all three reviews simultaneous, which would produce more information after one week but may be inefficient if the author made a mistake that used up all 3 reviews correcting the same mistake.)

  3. After the three reviews, authors will independently submit to scholarly journals with the goal that the journal process goes much faster due to this peer pre-review process.

  4. If requested by grantees, Social Science One will work with scholarly editors to enable the editors to use our peer pre-reviews to speed their decision making, in some cases eliminating some or all of their normal peer review process altogether. If the editors agree to confidentiality, Social Science One will:

    1. Privately certify to the editors how the reviews were conducted, the exact version of the paper that was reviewed, the timing and conditions, etc.

    2. Share  the peer pre-reviewers' identities with the journal editors (but not the author), so that these pre-reviews are more valuable to the editors and prompt the editors to be more likely to rely on pre-reviews and speed time to publication decision even more.