This year we tried something new to get feedback from attendees this year. After a talk you would put little plastic bingo chips into one of three buckets to vote. We also have a talk interest system on the schedule where we get a count on what talks people are selecting. I captured the data just before and after the conference along with other metrics. All this data I have compiled into a massive raw data dump on google docs. It’s a firehose of information, you have been warned. Check the second and third sheets for details on the data. I would like to boil this data down into something useful. More on that below.
History
It was started by a suggestion by Bruce Eckle a few days before the conference. He mentioned a system from another conference where attendees could put slips of colored paper into a basket at the end of the talk to indicate what they thought (red=bad, yellow=ok, green=good). Yarko suggested using marbles, and then Ted and I ordered had found and ordered 108 tubs of colored bingo chips (clearing out the supply). It was at this point that Yarko pointed out that 3 colored buckets and a scale would be simpler, faster, and much, much easier. It is taking me longer to type this than it took for all that to happen… In retrospect we moved a bit too fast on the idea.
Boiling it Down
There are a number of issues with the data. There were two occurances of ballat stuffing which are marked with comments on the spreadsheet. Ignoring those, and some cancellations. The biggest issue is that people vote with their feet at PyCon. If you dont like a talk, you go to a different talk. The IRC channel is filled with people more than willing to inform you of how the other talks are going. Some celebrity speakers can draw people from other talks. There is open space and plenty of other things to do. The raw data includes talk interest calculated before and after the conference from the schedule app. These correlate (@ about 97%) with actual attendance. How that works out is a post by its self as there are finer grained details.
I am thinking of finding corrilations within the data and then using that to compute weights to be used against the voting counts. Then using that to discount the green votes by the attendance and yellow and red votes. The end result will be quite noisy and most of the talks will fall within the delta error. Many of the talks will fall above the delta error and we can use those for determining which speakers to give invited talks and to influence the program commitee decisions next year. This information would also be valuable in conjunction with the online and printes survey information (to come later) for speakers.
If someone would like to help me with this I would be ever grateful. I love working on stuff like this but we need to start working on the 2010 stuff yesturday (actually some people have been working on PyCon 2010 Atlanta for months).
