Proposal Grading Rubric¶
This rubric is was developed during the 2020 proposal review period. Reviewing proposals is very challenging, we have lots of applicants that do great work and needed to find a way to quantify their contributions. We use this rubric to do so.
Grading Rubric¶
Grading Rubric Explanation¶
Explains why each field was chosen for the rubric. Some fields are subjective, others are objective. Each mentor will fill out a copy of the rubric for the subjective portion of the rubric. One mentor will count up relevant statistics to fill out the objective portion of the rubric. Once all proposals have been reviewed, the mentors rubrics will be averaged for each proposal. And the final score will be calculated for each proposal by tallying up all the points in the averaged subjective rubrics and the objective rubric. The proposals will then be ranked from highest to lowest. Final scoring will happen two days before slot requests are due to give students the most time possible to get more contributions in, get better at debugging, show self direction, and engage with the community.
Goals in Proposal¶
The project proposed should have a clear value to the project’s community. A high score here means if the project is selected (and subsequently completed), users would clearly benefit from it.
Completeness of Proposal¶
The proposal should contain enough details for it to be clear that the student knows how to complete the goals they’ve outlined. More detail beyond proving they know the basics of what needs to be done is better. If a student proposed something but was short on detail, their contributions to the project should be taken into account. If their proposal lacks detail on implementation of something similar to what they’ve already done several times, it’s not necessary that they re-explain in detail in the proposal, their contributions speak for themselves.
Time Commitment¶
Google expects students to spend 30+ hours a week working on their proposed project / the project they are contributing to (assuming they’ve completed their proposed project). Students vary in skill level. As such, mentors should take their interactions with students into account as well as previous work a student has completed to estimate if the student will be able to complete the outlined proposal in the allotted time. Mentors should also think about students previous work and how long it seemed like they spent working on previous pull requests to estimate how long each action item in the proposal will take the student. Using this estimate, mentors should respond in the rubric with their thoughts on how many hours a week they think the student will be spending working on their proposed project to complete it.
Engagement with Community¶
Student engagement with the community is key to their success, the success of future GSoC applicants, and the project. We cannot effectively collaborate to build open source software if we don’t talk to each other. The best outcome of GSoC is not only the completion of the student’s project, but the student becoming a mentor for other students or new community members. A high score here shows active participation in the community and that the students actions show them trending towards becoming a mentor / maintainer in this organization or another someday.
Mentor Vote¶
Every mentor should vote for one proposal which is their personal favorite. Each mentor has different aspects about the project (DFFML) that are important to them (maintainers of different subsystems, etc.). Since the mentors will be advising students, it’s important that they believe in the projects they are guiding. This is how they signal what they believe is important or cool.
Self Directed¶
While it’s always okay to ask what to do, it’s better to look at the list of things to do and pick one. Open source projects put a lot of effort into project management and organization of work items so that everyone knows what the open bugs are on the project and what the planned enhancements are. Everything is done in public because anything is accessible to anyone on the internet and it’s important that we all stay on the same page to effectively collaborate. Self directed students follow contribution guidelines and communicate their intentions to work on issues within the posted issue rather than other channels of communication which don’t allow others to see the status of what’s being worked on. This keeps the community in sync so there is no duplication of work. In some projects (like DFFML) issues are labeled with the estimated amount of work they will take to complete. Students who take on larger issues will spend more time working on the problem on their own and less time with a mentor stepping them through what to do. This is a measure of the students resourcefulness and tenacity for solving difficult problems.
Debugging¶
The act of working on any problem, programming in particular, is an exercise in figuring out what should be done, what has been done, and what’s the current problem that is keeping what has been done from being what should be done. This loop of fail, fix, repeat, is debugging. Students should be actively testing their contributions. When those tests fail they need to use critical thinking to determine if what they’ve done is working or not. If it’s not working they need to figure out what the next step would be in fixing it, and try that. If they can’t figure it out, then they should ask for help. A student that asks for help any time there is an issue without trying to figure out what the issue is before asking for help is not putting in enough effort. In addition, tests have to pass in the continuous integration (CI) environment. If the test doesn’t pass in CI, then the student should be performing the debugging process looking at what’s wrong in the CI environment. They should not say that they don’t know what is wrong because it works for them. Feigned ignorance of failing tests does not excuse their failure.
Contributions¶
Some open source projects (like DFFML) label issues with the estimated amount of work they will take to complete. Larger contributions, or those associated with larger issues, receive more points than smaller contributions. The goal here is to reward students who have put in more effort leading up to GSoC. They’ve worked hard to learn about the project they wrote a proposal for, and have worked hard to make that project better.