The final selection of the presentations for the FOSS4G has always been a controversy. Reviewing so many abstracts takes a lot of effort, so it’s not possible to tell the rejected presenters in detail why their abstract wasn’t accepted. Though it’s possible to be transparent about the process that was applied. That’s what this blog post is about.
Let’s start with some numbers. There were 287 abstracts submitted, 176 were accepted. There was a community review and a separate review by the program committee. Both reviews were blind, without knowing the author’s name.
Next the program committe met several times using a Google Docs spreadsheet which had the community rank, reviewer’s rank and the submitted details in it. This time it was including the author’s name.
The selection process
Step 1: Get rid of the lowest ranked ones
We were picking a number to elminitate the abstracts that had *both* a low rank (less thank 175) by the community and by the reviewers. This way we got the numbers down by 60, to around 230.
Step 2: Limit presentations by author
The abstracts were sorted and we went trough them to limit the number of presentations to a maximum of two per person (with exceptions: some people are so active in the FOSS4G community, that they represent so many different projects, so that it make sense to have them give more than two talks. More on that later). We removed the lowest ranked ones, taking the community as well as the reviewers vote into account. This brought us down by 10 to around 220.
Step 3: Go through the low ranked ones
We sorted the abstracts by the community ranking and went from the bottom up on a one by one, looking at the ones with a low ranking from the reviewer’s first. This got the count down by 20 to around 200.
Step 4: Limit presentations by author again
Going through all the abstracts that are by authors that have more than one talk. Decided on a case to case basis on whether to kick one out, to ask the author to combine the talks, or to leave the two abstracts. Some authors still had more than two abstracts, those were also highly ranked. Hence we decided to make them invited talks.
After this we had only a few people with two talks and no one with more than two talks. This lowered the number by 20 to around 180.
Step 5: Group talks by topic
At the end of the first meeting, we got the count down to a number that would make things roughly fit into the schedule. The task until the next meeting was to group the tentatively accepted talks by topic, so that the talks can be put into the schedule.
Step 6: Finding/discussing topics and wait list
In the second meeting we were trying to find common topics for the talks to put them into slots. We ended with several categories and were sometimes even able to find three related talks that will make it into one slot. During this phase we also identified talks that didn’t really fit into our program. Last thing for the meeting was the preparation of a waiting list in case presenters can’t make it to the conference.
We ended up having the 176 talks in 26 categories and 20 presentations on the waiting list.
Impact of the community vote
The final program compared to the community voting is probably the most controversial topic. Here’s the result: when you remove the presentations that weren’t accepted, due to being from the same author, only 17 presentations from the top 176 community voted ones were rejected. That’s a 90% overlap between the community vote and the final program.
The committee tended to defer to the community vote when the appropriateness of a proposal was in doubt. When the committee made decisions that were contrary to the community voting, it was usually for one of a three reasons:
- Authors with numerous proposals
- Proposals that were felt to be very similar to other, higher-ranked, proposals
- Proposals that covered topics the committee felt were neglected, or were up-and-coming topics that deserved attention
