GSoC Proposal Review

For reviewing Google Summer of Code proposals, all mentors are involved. Since 2020, we follow the review process outlined below. All communications regarding proposal review and selection are conducted in private using Discord (in 2020 and 2021 we used the @gsoc-mentors team discussion (in private mode)).

MDAnalysis typically receives on the order of 20 proposals. Not all are valid (e.g., not everyone read our requirements and has a PR merged). Invalid proposals are removed from consideration by org admins.

Valid proposals are reviewed similar to what is done on grants panels (e.g., at the NIH or NSF) during a teleconference. Budget about 15 minutes time per proposal. A week before the teleconference, proposals are assigned to primary and secondary reviewers (see below).

A Google spreadsheet with students and proposals is prepared and shared with all reviewers so that everyone can edit scores in real time.

Reviewers

1. We have one primary reviewer per proposal who
   • introduces it,
   • summarizes pros and cons.
   • presents own numerical evaluation (see Criteria below)
2. At least one secondary reviewer who has read it and adds to the primary's discussion.
   • presents own numerical evaluation
3. Everybody else then discusses and ranks the proposal (see Criteria below)

Criteria

scale: 10: best; 1: worst

• significance (Does the proposal solve an important problem for us, are the objectives clear?)
• student (Does the student have the right background and skills, will they be able to learn skills they don't have but need for the project, are they teachable?)
• approach (Did the student describe a feasible path towards achieving the objectives?)
• interaction with us (How did they communicate with us, both during working on their PR and during proposal writing?)

We are using a weighted average for the above scores:

score = 0.1 * significance + 0.2 * student + 0.4 * approach + 0.3 * interaction

with the emphasis on metrics that might be important for completing the project successfully.

Ranking

Once all proposals are scored, we perform the final ranking. Proposals are not automatically ranked by score; each proposal's and student's individual merits in relationship to our needs are taken into account, especially when average scores are close or reviewers have very different opinions.

Proposals are

• ranked from top to bottom
• assigned a primary and at least one secondary mentor

Based on this list, slots are requested.

Proposal acceptance

Slots are filled starting with the top-ranked proposal, taking into account mentor availability.