University of Wisconsin–Madison

The Project

The Open Peer Review project is funded by the National Science Foundation (NSF # 1747445 and NSF#1760092). Our goals are simple. We want to change norms about sharing data from scientific grant peer review processes by inviting the scientific community to donate their grant reviews (i.e., critiques and scores) to our national de-identified repository.

  • Why study the peer review processes

    Now more than ever is the time to study scientific grant peer review. Federal funds have not adequately increased to match growing numbers of grant applicants, leaving funding rates at an all-time low. Both explicit and implicit bias can influence peer reviews’ decision-making, differentially advantaging applicants in unfair ways. To ensure the best science gets funded, agencies, applicants, and reviewers must work together to study peer review processes and develop interventions to help minimize bias.

  • Why should you participate

    This project will help NSF to fulfill its mission to advance scientific knowledge and ensure the U.S. scientific enterprise by automating a novel method for collecting peer review outcomes from federal funding agencies and by maintaining large de-identified datasets, which will be available to the scientific community for analyses of the reliability, validity, and fairness of review systems. With increased access to peer review outcomes, the scientific community will have the opportunity to help agencies like NSF evaluate their review processes, ensure they target the best science, and reduce the influence of unintentional bias.

  • How can you help

    Donate unfunded and funded summary statements from NIH from the following award types:

    R01 (Type 1), R01 renewals (Type 2), R21, K01, K08, K23

    Complete the demographic survey

  • Our long term goals

    Intensive study and refinement of federal agencies’ review processes will help to ensure that U.S. tax dollars are allocated in ways that bring the broadest benefit to society. It will also expedite much needed research on identifying and mitigating sources of bias and improving reliability among reviewers. Importantly, results may help to broaden participation in the scientific workforce, because individuals from historically underrepresented groups (e.g., women, racial/ethnic minorities) appear to face disadvantage in review processes, which precludes their ability to persist and advance in science, technology, engineering, mathematics, and medical (STEMM) careers. This project will help to illuminate targets for intervention that may help “level the playing field” and ensure minority scientists have equal opportunity to succeed in research careers.

  • How does this relate to diversity

    Currently, data collection is focused on the National Institutes of Health (NIH), which funds most biomedical, behavioral, and clinical research in the U.S., primarily through its R01 grant mechanism. Racial/ethnic minority and female scientists have lower R01 (Ginther et al. Science, 333(6045):1015-1019; 2011); and R01 renewal award rates (NIH, 2017Pohlhaus et al., Academic Medicine 86(6): 759-767; 2011), respectively, even after first obtaining Mentored Career Development awards (e.g., K01, K08, K23) – traditional precursors to R01s (NIH, 2011). R01s are critical for career advancement in biomedicine, so funding disparities contribute to the disproportionate loss of racial/ethnic minority and female scientists from biomedical research careers. Scientists from these groups are more likely to study health issues of underserved populations, and to shift research paradigms in innovative ways (Valantine & Collins PNAS, 112(40): 12240-12242; 2015). To address health disparities and to ensure the U.S. remains competitive in the global economy for science and technology, therefore, NIH funding disparities must be resolved and we need your help!

You are eligible to participate because you are a scientist or faculty member in a science, technology, engineering, mathematics, or medical field and may have applied for research funding from federal agencies, such as the National Institutes of Health (NIH).

The Open Peer Review project is funded by the National Science Foundation (NSF # 1747445). Our goals are simple. We want to change norms about sharing data from scientific grant peer review processes by inviting the scientific community to donate their grant reviews (i.e., critiques and scores) to our national de-identified repository.

Now more than ever is the time to study scientific grant peer review. Federal funds have not adequately increased to match growing numbers of grant applicants, leaving funding rates at an all-time low. Both explicit and implicit bias can influence peer reviews’ decision-making, differentially advantaging applicants in unfair ways. To ensure the best science gets funded, agencies, applicants, and reviewers must work together to study peer review processes and develop interventions to help minimize bias.

One of the greatest barriers to studying scientific grant peer review is accessing data, and most agencies hands are tied because they must protect the identities of applicants and reviewers.

Our prior studies have shown that most federal grant applicants do not view their critiques and scores as sensitive, and are willing to share them for research processes, especially if they are de-identified of investigator and institutional information.

Currently, data collection is focused on the National Institutes of Health (NIH), which funds most biomedical, behavioral, and clinical research in the U.S., primarily through its R01 grant mechanism. Racial/ethnic minority and female scientists have lower R01 (Ginther et al. Science, 333(6045):1015-1019; 2011); and R01 renewal award rates (NIH, 2017; Pohlhaus et al., Academic Medicine 86(6): 759-767; 2011), respectively, even after first obtaining Mentored Career Development awards (e.g., K01, K08, K23) – traditional precursors to R01s (NIH, 2011). R01s are critical for career advancement in biomedicine, so funding disparities contribute to the disproportionate loss of racial/ethnic minority and female scientists from biomedical research careers. Scientists from these groups are more likely to study health issues of underserved populations, and to shift research paradigms in innovative ways (Valantine & Collins PNAS, 112(40): 12240-12242; 2015). To address health disparities and to ensure the U.S. remains competitive in the global economy for science and technology, therefore, NIH funding disparities must be resolved and we need you help!

  • Our previous work suggests that combined text and scoring analysis can provide evidence about the extent to which unconscious race and gender bias are operating in NIH peer review to differently advantage applications based on demographics.
  • Please donate your NIH Summary Statements (i.e., the files that contain scores and critiques) for all your UNFUNDED and FUNDED applications since 2010 of the following award types:
    • R01 (Type 1)
    • R01 renewals (Type 2)
    • R21
    • K01
    • K08
    • K23
  • Your Summary Statement(s) will be converted to txt files and de-identified. The original identifiable pdf files will be destroyed. A program will go through each txt file and pull all scores and critiques into a csv file, which can be used for text and data mining.
  • We will also ask you to fill out a demographic survey, which will be stored as de-identified information linked to data from your de-identified Summary Statement(s). This information is critical for our analyses, and we provide every protection possible to our participants.

Follow this link to donate your Summary Statements and complete a survey about your demographics.

This project is supported by:

                               NSF EAGER – Award Number: 1747445  and NSF Award – Award Number: 1760092