If you are committed to open science, there are fairly limited options for reviewing papers. Basically your only option is to review for journals which support and enforce open science principles. You are then trusting the journal to make sure the author makes their data and code available, and also publishes the article open access. Moreover, you are forced to decline to review interesting papers because they are not submitted to a limited number of journals supporting open science, or because the authors couldn’t afford the article processing charge and submitted to a closed-access journal. This approach provides no feedback mechanisms to the authors of these papers that you would have been happy to review their paper had they committed to making their article, code and data openly accessible to the community. It provides no incentive to authors to be more open.
We wanted to create an alternative, and this is where the idea of open-science agreements came from. When you are invited to review a paper, you can visit http://academickarma.org/reviewagreement to specify (anonymously) what you expect from the authors in terms of openness before you agree to review their paper. You also specify how open you plan to make your review, and how long you expect it will take you to complete the review. Rather than asking the authors to pay to publish open-access, the reviewer can ask that both the submitted and revised manuscripts are uploaded to a preprint server. The authors can suggest modifications and explain if they are unable to satisfy all of the reviewers expectations. If an agreement is reached , then the reviewer can review the paper with the knowledge that their free labour is supporting resources (code, data, manuscript) which can be re-used by everyone.
More information on how open science agreements work is provided here.