A dive into needs and wants and whys.

Also see:

• Introduction – Lets make a bug tracker!
• Background – It’s not really a bug tracker!

Time to get to some use cases, requirements, needs, wants, and technology. As mentioned in the last post, I need something to replace the features we currently have in Lotus Notes. While this is a project inspired by issues at work, and I hope it will be used there, the actual needs and requirements are coming from me. Not everyone in my group would agree with what I am calling requirements. Some of my requirements are there because of specific pain points I am having, but which do not really bother others. But hey, it’s my project, so they don’t get a say.

With that, it is time for a

Disclaimer: The postings on this site are my own and don’t necessarily represent my employer’s positions, strategies or opinions. Opinions and ideas expressed here are mine and mine alone.

Objectives

The pr1mary objectives are:

• Increase visibility with ease of access (web interfaces or simple app)
• Integrate with other existing systems (Perforce, Plone, Active Directory, release processes)
• Preserve and or migrate sixteen years of history (old bugs, api discussions, etc.)
• Support existing workflows (fast ramp-up for users)
• Minimal impact on schedules (get it up and running without anyone noticing until it is done… hi boss….)

That’s about it really.

High Level Functionality

Quick summaries of the functionality we will dive into.

• Issue/Bug Management – Our needs are very simple, but specific. Multi-Project, RSS, large files, and read/unread markers being chief among them. The biggest issue here is how we manage the forks. Each bug/issue will have a status in multiple forks. There are multiple ‘active’ forks which need to have their status set. As new forks are made, the system needs to know about them. Archived forks are not set on the bug. A bug is not fully closed until it is closed on all forks. Very few issue management systems do this out of the box, though many support similar work flows.
• API/Code Review - Nicely color coded diff of changes to the API with the ability to make comments on the changes, and have multiple versions of the multi-file changes. This is key for the API review. General code review is secondary. This need not be fully integrated into Perforce via triggers. That option would be nice, but for the API review, it would be better to submit a changeset via some script (*cough* upload.py. gee… what ever system could I be thinking of for this….) Essentially we need a way to review patches which are not checked in anywhere. It would be nice to review changes which are pending or already submitted as well, but that is a want, not a need; we have that functionality in another trigger system already.
• Discussion Threads – Similar to a bulletin board, but with thread and read/unread requirements. The key features from Lotus Notes are the overview listing with read/unread status. The ‘topic’ can be changed and is modified over time with a change history. Underneath in the main list view are threaded responses. Everything has read/unread markers. Think of a threaded e-mail archive, where the initial e-mail can be edited with a change history. BBS systems come close but miss the mark on the key requirements.
• Static Release HTML Help/Docs – (sideline work) Technically not something that is in Notes and already exists internally, but we need to integrate with it cleanly so mentioning it. Adding an app to smartly serve up static files are set locations would be simple enough to add and would replace some fancy apache rewrite rules.
• Interview Management – (bonus work) special discussion system for managing resume’s, code problem solutions, etc. Has special permission requirements, and voting (+1, +0, 0, -0, -1)
• Machine Management - (bonus work) very very simple app for tracking who has what hardware specs on their machines. Not asset management. Just useful for when we have budget to get new hardware, or when a bug appears to be hardware specific.

Some of this could be moved into Plone, or just abandoned. Much of the project management occurs in Plone and integrates across projects. Different projects use different tools, but everyone integrates at the Plone level. One restriction on the Plone side is, because many groups are using it, there is a very high barrier to adding new functionality or special customization. Also we do not have the option to go with a cloud hosted solution like some groups, due to our access to medical records. Just the access to such records means we can not risk it.

Issue/Bug Management

Requirements:

• Active Directory Integration
• Perforce Integration (very minimal)
• P4M Fork integration (one bug, many forks)
• P4M Version integration (is the format valid)
• RSS Feed (cheap Plone Integration)
• Large file support
• Milti-Project
• Read/Unread Marking
• Filters (open, closed, owned by, etc.)
• Searchable

Nice to have:

• Strong Perforce Integration, Checkin’s and Changesets (i.e. see the code from the issues)
• Strong P4M Fork/Version integration (does a fork/version exist, active forks, etc.)
• Smart links (turn @34245 into a link to the changeset, etc)
• Colorized source code diffs
• Nice WYSIWYG Editing instead of some wiki markup
• Custom reports, queries.

API/Code Review

There is a different between how we handle the API review than other code reviews. We have a requirements, and proposals. This is can be a many-to-many relationship, but usually is just one-to-one, or occasionally one requirement to many implementation proposals. Sometimes a bug will be referenced as it may require an API change to fix. Often it is a client which will submit a requirements document, which then gets refined.

Requirements:

• Active Directory Integration
• Multi-file changesets
• Colorized changes
• Full and partial change context
• Multiple revisions of a changeset (i.e. one proposal, multiple changesets)
• Requirements document support w/ linkage to proposal
• Change history on requirements document
• Clear history on the changes to the item being reviewed
• Good comment system (duh!)
• Integration with bug/issue system (may just be a hand generated link)
• RSS

Nice to have:

• P4M Integration (submit a changeset from P4M)
• Perforce Integration (Code review of already submitted changes)
• Nice WYSIWYG Editing instead of some wiki markup
• Custom reports, queries.
• Read/Unread markers (this might be moved to a requirement)
• Searchable

Discussion Threads

Discussion threads are interesting in that they closely resemble the functionality in a BBS system, but with some very important differences. The initial topic post is edited with a change history, and then there are threaded comments below. The threaded comments are their own posts. The best way to think about it is your standard mailman archive thread view, but where the author of the initial e-mail can go back and edit their post, and the responses can have different titles. Each entry has it’s own read/unread state. This could be achieved in Plone with a single page with change history and threaded comments except that the comments are at the bottom of the post, do not have read/unread state, and can not be listed as part of an overview tree/list. So they are really nothing like that Which is why it is hard to find something which supports this mode of discussion we are so used to in Notes.

Requirements:

• Active Directory Integration (why do I keep listing this?)
• Multiple ‘boards’ or whatever (multiple databases in Noted language… just call it Multi-Project…)
• Topic listing with responses all in one list, with thread indentation (needs a picture to describe)
• Change history on Topic level posts
• RSS
• Read/Unread Marking
• File Attachments
• Filters (by date, person, etc)

Nice to have:

• Smart links (turn @34245 into a link to the changeset, etc)
• Colorized source code diffs
• Nice WYSIWYG Editing instead of some wiki markup
• Custom queries
• Searchable

Interview Management

This is a fun one. It is just a very simple Notes discussion database (actually all of these except the bug management is just a simple notes discussion database), with some extensions for voting. When a resume is approved via triage, an entry is added with it attached. We vote on resume’s with a comment. We decide who gets a phone call from these votes, and then people are assigned to make the calls. There is a ‘response’ made to the resume with notes from the phone interview. If the phone interview goes well we send out a programming test. We wash and test the response program. It gets posted unwashed as a response to the resume. The washed version with comments and notes is posted as a response to the unwashed version. Then bring the person in for the real full group interview. Then we try to hire that person. Whenever we are done with the resume, it is ‘archived’ with these notes where only the manager can see it. Very simple and a quick app to write.

Requirements:

• Active Directory Integration (blah blah blah…)
• File attachments
• Ability to ‘archive’ entries to be seen only by manager
• Voting by group members (+1, +0, 0 -0, -1) with comment.
• Status (voting, needs phone interview, waiting for program, etc…)
• Assign person to phone interview/code review
• Read/Unread Marking (see a theme here?)
• Filters (those which I have not voted on yet, needs votes, needs votes by person, a few others)

Nice to have:

• Multiple repositories for different groups (call it Multi-Project)
• Colorized source code?
• Nice WYSIWYG Editing instead of some wiki markup
• Searchable

Machine Management

It’s just a simple table with change history!!! Bah, create a Plone page and be done with it! Why is this still in Notes?

The Big Picture

Really everything currently in Lotus Notes, sans the bug tracker, is based on the basic Notes discussion database. It is best to look at all of these existing pieces of functionality and see if the lines drawn between them are real or artificial. What I mean by that is, much of the code commentary need not be done in discussion threads, but instead as part of a generic code review application. The distinction between API review and code review is important due to who gets input and when, but not much beyond that. The discussion threads are a catch all for many different types of design resolution that do not fit in the API discussion, or the bug tracker. Plone is most likely the best place for much of what is currently in the discussion database, with the code, and file format stuff moving to a code review app. The machine management database is nothing but a poorly implemented Plone page with a table. The real things which are needed are a strong issue tracker, a powerful and configurable code review tool, and a very simple voting app for resume’s (another voting app? *sigh*).

So next up is evaluating existing technologies to figure out which fit best, while taking into account both the background and the big picture. Just in time for me to go back to work in a few hours too… So much for getting it done over vacation. Failure #1 took long enough. 

Now it is time to give the concrete context in which this project is being done. This could be considered the pre-requirements. It is important to know how your project is going to be used and the environment it is going to be used in. If you want your project to succeed, this information is crucial. this is where you find out what is really needed over what is requested; especially when the person declaring what is wanted is yourself.

This is going to be a very long post, but I will be touching on almost every sentence later on as this all needs to be taken into consideration. It is crucial to looking at these big picture issues when designing some tool or application which will be used in that context. To properly understand the why behind decisions I later make, I would rather dump it all here and refer back later. I hope it will be interesting to some at least.

As the background for this project is going to delve into the processes and development procedures at my day job, it is time for a,

Disclaimer:

The postings on this site are my own and don’t necessarily represent my employer’s positions, strategies or opinions. Opinions and ideas expressed here are mine and mine alone.

Who we are

My title is ‘Principal Research Engineer, MREC’. “MREC” is the modular speech recognizer project. We are a development group within research, in charge of productizing the algorithms and features needed by our extensive research teams, and deliver a core speech recognizer. We are a small team of 8 developers. There are many internal clients with technology built on top, and we are not the only engine in the company. Everyone on the team wears many different hats. We are all developers, researchers, operations engineers, QA engineers, release engineers, technologists, and much more. Everyone works on everything, down to the build, release, deployment, and source code management tools. No one is irreplaceable.

How we Work

There has been a wave of ‘agile’ going through the company which I personally find both invigorating and frustrating. At issue here is that we are really talking about ‘agile-but‘ at best. What is frustrating is that our team has been agile sense before agile existed. We release often, and not all releases are used by all our clients. Some are only really used by our team. Most often releases are only used by our researchers, and never see a product. A release is made either because we would release on a date (irregardless of what is in the release) or because a feature or set of features were just completed. It has always been that way.

We do mainline development with continual integration. That is we have one single branch which all the work is done. When a major release occurs we fork the project, and the newly created fork is a maintenance fork which only has bug fixes. Any bug fixes are first implemented in the mainline and then back-ported to the forks which we determine need it. We try to only keep one or two forks live; which means sometimes forcing clients to upgrade to a newer version even when they only think they care about a single bug affecting them.

Notice I have not mentioned branches or branch development. We don’t do that. Check in, and check in often. Update with the mainline multiple times a day as needed. All tests must pass before checking in. Every bug is really two bugs, the bug and the missing test for it, both must be fixed to consider the issue resolved. This may sound like it slows down development, but just the opposite is true, and I would never work any other way now. We do have the ability to branch and work in isolation, but we prefer to keep that type of thing very short, and to be honest I can’t think of a time it has been needed. The point is not that you need to integrate with other people changes, but that you need to get your changes in to the mainline immediately so that everyone else can integrate with you.

There are code reviews. We only implement what is needed to solve the underlying problem or request. We do not implement what is requested, but what the requester needs, and rarely go beyond that. We do not implement features that are not used. We love to remove code and unused features. Over the past sixteen years we have removed three times more code than the current size of the project. We have competitions to see who removed the most code. That is the only code metric we look at with any seriousness. Our client facing API is reviewed by the entire team and all clients, and is approved before checked in. The documentation for the API is in the API headers. We bikeshed on the API endlessly; or at least it sometimes feels that way.

None of this should look surprising, revolutionary, or out of the norm. All the major open source projects I know of work this way; more or less.

Communication is the Problem

As I said, there has been this wave of ‘agile’ in the company. Recently someone said to me, “I saw this months sprint document for MREC. It is great to see that MREC is going agile too!”; and I bite my tongue. I want to say, “You keep using those words. I do not think they mean what you think they mean.” When a term like ‘sprint’ is used is does not have the meaning I expect it to have. Let me be clear here: the problem is me, not everyone else. The problem is not inherit in the systems, but in the communication. My communication and the communication between our group and the rest of the company.

Our group has been operating somewhat like a black box. Different clients would file requests and our manager would work with the clients and put together a general schedule for getting features in, and re-prioritize as needed. This is a group effort to figure out what needs to get done and in roughly what order. Our clients do not talk much to each other for prioritizing our work, as we do the prioritizing. The end result is, while we have advertised what we are working on, and what goes into a release, these things have been very hard for our clients to track. They know when a feature they need is in a release, but then they have to accept all the other features for all the other clients, which they have not been tracking. One could argue that this is not our groups fault, as we do make all this information available in one form or another; but that is not really fair and as we will see ‘one way or another’ hides a multitude of sins.

Communication is the Solution

Much of this is old news and we have changed much in what it appears we are doing. In truth we have not changed the way we develop at all. We instead are exposing what we are doing and allowing our clients more control over our priorities. The end result is that while there is no real change in our priorities, our clients understand them much better and feel like they can predict things better using agreed upon terms like ‘sprint’. In truth it is just a change in perception; instead of not reading internal web pages and e-mails with text attachments, they are not attending meetings, not not reading different internal web pages, and not reading different e-mails with excel attachments. But these are meetings, pages, and e-mails they asked for and we designed together, verses things our group dictated; which makes all the difference in the world. Yes, I am being hyperbolic.

Even with this added communication, which is real and effective, there are still problems. They center around our tools and environment. We are in the late parts of migrating to new systems, but all this has done is highlight the flaws and weaknesses in the systems we have yet to migrate.

The Environment

The different teams have different development styles which best fit their needs. Same goes for the tools used. The source code management we use is an in-house wrapper on Perforce called P4M. This is open sourced as part of the DevTools project; which is woefully out of date and we will be releasing a new version of soon. We have multiple projects hosted on a single server. Our team has four projects including MREC, and other teams have projects as well. P4M enforces a directory layout including forking and branching. We choose Perforce over SVN or a DVCS for many reasons. We need to deal with huge files (>2Gig) and a basic install takes up >32Gig. The MREC tree alone has 300GB of history. We need to be very careful of permissions. Our engine is used in many medical transcription applications, so we must be very careful. We try never to check in unwashed data which might contain any PII. In the off chance that some does get through we need strict centrally controlled permissions. At any time we need to know who has what on which machines. For patent and litigation reasons we need everything centrally managed with strict backup rules. Perforce really is the only affordable system out there which can handle this. The entire 16 year history of the project is maintained and with proper dates and attribution.

Our project planning is managed external to the source code management, and the bug tracking and issue management. This at first might sound odd, but with so many internal clients, the priorities are often changing. In our issue management system, we have 3 severity levels (low, med, high), and a fourth level called ‘request’, for all features and improvements. Requests are automatically lower than any bugs. This has caused some friction with client, especially research. The problem is, the person filing the request really has no clue what the real priority is for a request. That is unless they are a project manager and have consulted all our other client product managers, and want to take on the task of updating the priority over time. As such the real priorities are managed in another system which links back. Known bugs on the other hand are always more important to fix than a new feature or improvement. This is a royal pain to try to communicate, and still results in heated discussions; we will get to this in a later post.

Releases are made to network directories which are accessible from the same path on all our grids. There is a ‘current’ symlink for the latest supported version, the version is always increasing, and the directories are named with the version number. Each checkin to a codebase get’s its own unique increasing version number (managed by P4M). The release includes built static html documentation. This documentation is served up by an apache instance, or can just be accessed directly. We are migrating away from a hodge-podge of doxygen, pydoc, and other inhouse hacks over to a standardized sphinxdoc system. this includes taking the help from our API headers and generating sphinxdoc which is cross references with our Python interface. The overarching project planning, and release information is in a plone instance which is used by all of research and development. This is all magically cross linked and searchable; or will be once the sphinx stuff is done, some of which will be tackled by this hear project. Our sphinx extensions will be released as part of DevTools, and some of the release exposure stuff will be dealt with this new project.

Lotus Notes

There are some pain points which have been hitting me over and over again now that the other parts are so much better and fully integrated. Specifically the way we review changes to our API, discuss architecture and format changes, and issue tracking. All three are managed in a 16 year old Lotus Notes system which has not changed much in 10 years. Very few groups in the company still use notes, and most can not be bothered to install the client. Most have other tools for these things which have web interfaces, or similarly, are causing communication problems. Requiring a tool like Lotus Notes does not sound like it would be much of an issue at first, especially as Perforce is required for code access. But when you realize that this includes a new e-mail address which is not integrated with your ‘real’ exchange address, and you need help to set it up to make sense, and that we have no IT staff which know or understand Notes, nor a support contract; heck for all the nice things we have in it, I don’t want it. There is 0 chance of getting a web interface as well. In short it is an isolated island which plays well with it’s self, and can be linked to externally with some contortions (on windows only, with specific OLE/ActiveX plugins), but does not play well with others; and playing well with others is the point.

I do want to give a shout out to Eric Ochieng who set up the Notes infrastructure years ago and which has been working flawlessly sense with no real support or maintenance.

What this project really is

This project really is not a bug tracker. What it really is, is something to replace all the functionality we currently have in Lotus Notes, with other things which integrate into all the other things we do. Part of that is indeed issue tracking, but focusing on just that would be a failure from the outset. We could go with something like CearCase+ClearQuest+Replication+Send-All-Our-Revenue-To-IBM, and spend months trying to get it all working, but that is just silly. There are a number of open source and pay for solutions which almost fill our needs. It is that ‘almost’ which is the problem. There is nothing out there which does the things we need, let alone cover the things we want. This project is to dive down into what the real needs are verses the wants. Find the solutions out there which come closest while integrating together and with what we already have. the pieces which are still missing, I will create from whole cloth of some type. Python is not a requirement, but I will be leaning towards it, as I am the only person on our team which knows Ruby or Java well enough to support those languages; we are allergic to .NET for reasons I will go into later.

Up next: The Big Picture – a dive into needs and wants and whys.

The point is not to watch as some ‘master’ weaves a new project and does things in the ‘best’ way. It is not for me to impart some special wisdom. Oh, no… The point is for me to fall on my face. Hard. Repeatedly. In full view of everyone. When I see something done right I rarely learn anything from it. The most revolutionary concepts often go completely unnoticed because they just work. And no one remembers things that just work; until they don’t. So this ‘Grand Experiment’, I hope, will fail many times over.

Another reason why I have not done this before now, is that I have not really had a project which will work with this concept. Oh I have some small utility projects (django-app-plugins chief among them) which desperately need my attention, but I am not actively using them for anything. I have found that when you work on a project like that without an actual site using them, something driving the development, you end up having to envision how you would like to use it. Then when an actual use case comes up it does not quite fit what you thought you might use it for. So this is going to be driven by real use cases for a mission critical application which will be used by a small development team.

How the blogging will work.

There will be an introduction (i.e. this post), background post, big picture post, and then breakdown posts. the breakdown posts will cover the work as I do it. There will be no overarching organization beyond whatever I have had time to do and get checked in. The breakdowns will cover the work which was done in the interim commits, and or new use cases. Each use case will also include research done on options on implementation details. Things which do not work will be highlighted. Even things which do work will not work perfectly. The entire reason for this project in the first place is because nothing in existence works well enough.

So why now?

Good question self. Well there has been a pain point at work which keeps coming up. I have a mandate to specifically NOT work on this. I also am on vacation, and have not done any programming for me. But I have seven days, a wake, a funeral, games night, home owners association management company migration, PyCon issues, three parties, two kids, and a none to happy wife. Did I mention that I do not have time to do this?

So why blog it at all?

Another good question self. Because doing this in public will produce better code. At the last Django Meetup it was commented by someone unnamed that they produce much better code when it is open sourced than when it is done for work, or just to scratch that itch. Everyone present agreed that it was the case for themselves as well. And finally, by doing it this way, it is clear that while this project may end up being used at work, it is done by me, on my own machines, on my own time, and is completely tangent to technology my company cares about. It is related to another piece of software which my group at work is already open sourcing. Oh and the entire falling on my face thing… that is just too much fun to pass up.

So what is it?

It’s a bug tracker.

Wait WAIT!!! come back… don’t stop reading!

The background and overview posts will go into the research behind all the alternatives, and this project will be more about leveraging existing technology, extending existing projects, writing the few apps which are needed to fill in little bits, and more than anything else, gluing it all together. The emphasis will be on the use cases and work flows which our group uses and dealing with less than enjoyable requirements. I do not even know if this will be used by my group at work in the end, but that is not really the point after all. There is going to be sphinx, django, jquery UI, rietveld, Perforce, DevTools, and much, much more. I was told specifically not to work on it; which has almost the same effect as telling me something is impossible.

The start of this polite rant was that he is the author of a commercial Unbuntu book, and he made a website to support the book. He was clear in the use of the trademark ownership, but did not get permission first from Unbuntu. During conversations with Connonical it was mentioned that he might have taken liberties, but that was just in passing. Instead of asking for permission (which we would most likely have been given) he removed the logo and the trademark in all places except where nominal use could be applied (i.e. the book title) and writes a blog post calling for the abolishment of the use of Trademarks in Open Source.  To me this is Trademark law working perfectly well for Open Source. If you are doing something commercial, the owner of the mark wants the ability to aprove it to make sure its not malicious. They are taking as light a hand a possible in it’s use in allowing non-commercial use freely without the need for seeking aproval (NOTE: you have to use it.. no really, read on)

There are a number of mental failings in that posting. First and foremost is that it forgets the lesson that the GPL and Open Source are reliant on the copyright system (and I would argue that even if there was not copyright that I would want it if only FOR the GPL which enforces the principals of giving back.) The argument is not to have any trademarks at all (as no other option is provided anywhere.) He calls out trademarks as being the antithesis of Open Source. You could literally replace ‘Trademark’ with ‘Copyright’ in his end section (with one minor exception.) But we need copyright law for the GPL to work; its better than the Public Domain. Just as with Copyright, it is the application which should be addressed, not the tool it’s self. Any ‘solution’ should use the trademark system to support Open Source.

What tends to happen is that open source companies have to walk a tightrope, and slightly strange rules on trademark get put in place. For example, Ubuntu is cool with community remixes using the trademark, but if you intend to make money from Ubuntu and want to include the word in your business title, you’re going to need permission. It’s not quite clear here how the former won’t dilute the Ubuntu brand, while the latter possibly will. The “protecting brand identity” argument falls apart almost immediately upon examination.

Um… then you do not have much experience in the real world. Would a company which called its self ‘The Unbuntu Corporation” and sell a product ‘Real Ubuntu’ based on a very old release or containing problems on purpose dilute the name? What about the ‘Python Corporation’? What about someone selling ‘Django 2.0′ (which turns out to be a fork based on 0.95). Would these things dilute those projects? I think they do. Allowing free use for non-commercialuse may have a risk of dilution, but as this is an Open Source project and crucial to the founding and flourishing of a community, more often than diluting, it helps build the brand. Those are the uses which you want to promote as being part of an Open Source community. These are real issues and real problems which we are facing today.

Here is the crux. If you choose not to use Trademark law to your benefit, others can use it as a weapon against you. If you do not actively protect your trademarks, you loose them, and you can be attacked. I actually am a little happy that there have been some recent issues with the use of the Python logo and trademark as it proved that they are important to have and protect.

The nature of Linux and open source in general is to encourage forking and splinter projects. That’s the basic freedom provided by the GNU Public License, and similar licenses. Some of the forks or splinter projects will be poor quality. Some will fail. But that’s just the way things work with Linux.

Ah, and here’s the rub. If a project forks into 2, or say 4 different projects, do all those projects have the same name? What if all the different linux distributions were just called ‘Linux’ without the qualifiers ‘RedHat’, ‘Fedora’, ‘Unbuntu’, etc. Branding is important. If you invent a better version of FireFox based off of the source code but with different options and behavior and code, then do you get to call it FireFox? Would you even want to? Would you want it to be confused with this ‘failed’ version? Wouldn’t you want to rebrand it to call it out as being better? Maybe IceWeasel? Yea I know I am going to get flamed. I agree that debacle was stupid. But I can understand how the problem came about and I do not see malice or ill intent as the author does:

Mozilla is even worse. If I create a new Linux distro, and include my own compiled Firefox binary, it’s unlikely I would be able to call the browser “Firefox”, or use the familiar fox logo, without getting permission from Mozilla. This could put me at a competitive disadvantage compared to other versions of Linux because my users would be using what appears to be unfamiliar software. It’s worth mentioning that Mozilla’s trademark rules also indicate they’re not terribly happy about the unofficial redistribution of their binaries, either, and would prefer it if they were the exclusive source.

But its not the same software, now is it? (well maybe not in this exact instance, but that is what the Trademark protection is supposed to be used for. To prevent a fork from being called the same ‘proper name’ as it is a fork.) Once again it is the use of Trademark law, just like copyright, that matters.

Is this how open source is supposed to work? Redistricted redistribution? Tight control on who can compile software and still be able to call it by its proper name?

Um.. yes and no. There is no restriction on the distrobution of the source code. Trademark has nothing to do with source code, it is just about the use of a word or words as an identifier. There is nothing stoping people from compiling software can calling it by its given name. It’s not about the ‘who’ but about the ‘what’; or at least it should be that way. If I compile firefox exactly like the Mozilla Foundataion does, then I can call it FireFox. If I change what could is compiled and packaged, I can not call that package ‘FireFox’. I do not agree with this decision myself, but I do understand the slippery slope they are trying to protect against.

So lets recap:

1. Trademark in and of it’s self is not the problem (the same way copyright is not), but how it’s applied.
2. If you do not use Trademark Law, it can and will be used against you (and can be usd to make you change your existing project from say ‘Phoenix’ to say ‘Firefox’ and pay fees for violation.)
3. Trademark law does not impact your ability to distrobute, change, fork or compile code.
4. Trademark law does impact what you can call something. It gives you the ability to say how a name can be used.
5. The improper use of Trademark Law could be a problem for Open Source, but the majority of the issues (and the one which started his rant) are actually good uses of Trademark Law for Open Source.
6. Preventing corporate and malicious use while promoting the non-corporate communities, and approved corporate use is good for Open Source.
7. Why not ASK if you can use the trademark on your website for your book promoting the project?

There is a part of me which would love to take about 20K and start to go all evil with respect to trademarks, taking Open Source names and preventing projects from calling themselves things. Forking projects and trademarking them for my own purpose and calling them the ‘REAL’ version. It is not about not Trademarking, but about comming up with good guidelines for its proper use in Open Source. It is a tight line one must walk, but walk it we must.

History

It was started by a suggestion by Bruce Eckle a few days before the conference. He mentioned a system from another conference where attendees could put slips of colored paper into a basket at the end of the talk to indicate what they thought (red=bad, yellow=ok, green=good). Yarko suggested using marbles, and then Ted and I ordered had found and ordered 108 tubs of colored bingo chips (clearing out the supply). It was at this point that Yarko pointed out that 3 colored buckets and a scale would be simpler, faster, and much, much easier. It is taking me longer to type this than it took for all that to happen… In retrospect we moved a bit too fast on the idea.

Boiling it Down

There are a number of issues with the data. There were two occurances of ballat stuffing which are marked with comments on the spreadsheet. Ignoring those, and some cancellations. The biggest issue is that people vote with their feet at PyCon. If you dont like a talk, you go to a different talk. The IRC channel is filled with people more than willing to inform you of how the other talks are going. Some celebrity speakers can draw people from other talks. There is open space and plenty of other things to do. The raw data includes talk interest calculated before and after the conference from the schedule app. These correlate (@ about 97%) with actual attendance. How that works out is a post by its self as there are finer grained details.

I am thinking of finding corrilations within the data and then using that to compute weights to be used against the voting counts. Then using that to discount the green votes by the attendance and yellow and red votes. The end result will be quite noisy and most of the talks will fall within the delta error. Many of the talks will fall above the delta error and we can use those for determining which speakers to give invited talks and to influence the program commitee decisions next year. This information would also be valuable in conjunction with the online and printes survey information (to come later) for speakers.

If someone would like to help me with this I would be ever grateful. I love working on stuff like this but we need to start working on the 2010 stuff yesturday (actually some people have been working on PyCon 2010 Atlanta for months).

Registration

Online registration is closed. The full count is not determined yet (due to sponsor passes, financial aid, and a few minor issues) but attendance will be around 850, but could be as high as 900. We had 400 people in ’06, 600 in ’07, and bearly broke 1K in ’08. With the dramatic downturn in the economy and corporations removing all travel and conference expenses, this is astounding! Initially we were planning on about 1200 (plan made back in April last year), and some believed halving that number would be a better estimate not long ago (well I made statements to that effect anyway).

The new registration system this year is fantastic (Granted anything would be better than last years mess). Massimo, Yarkot and Carl really out did them selves. A. M. K., Yarkot, Massimo, Carl, David, and Kurt have done an outstanding job managing all the issues and fielding attendee requests. Early on I was keeping track of issues which poped up on the reg list to make sure someone was taking care of them (but insanely thankful I was not doing it this year). I stopped after 3 weeks as I was just wasting my time. There were a few kinks here and there, but they were minor.

The Schedule

I never did a follow-up on the program committee work, mainly due to time and general program burnout. I think the schedule is fantastic. If you don’t like the way the talks are scheduled, you can blame me. I stole the job from the person whom was doing it and unilaterally put together the schedule you now see. It was a beast of a job. I tried to get the talks placed into tracks. A testing track, a core python track, etc. I also tried to make sure no two talks occuring at the same time would appeal to the same people. I also tried to balance the talk levels to ensure there was something for every level at a given time and that tracks progressed across the conference. That way you could go to a beginner talk on a subject on the morning and be ready for an intermediate talk on teh same subject in the evening or the next day. Against that backdrop there was also the requirement that anyone giving multiple talks would have those talks given on different days. Some speakers have 2 talks, none have more than that, and it is key to only give one core talk a day if you want the quality to be maintained. Also there was the problem of the talk lengths….. My first draft has some glaring errors in it which thankfully other noticed.

Last night I actually made it to the Boston Python Meetup. There 3 PyCon presentations were given and they were… encredible! I am sooo glad I was able to make it. At first I went because I wanted to see the talks so I could see other talks at Pycon during those times. I will be attending Jesse Noller’s talk again at PyCon. I know I will learn more by seeing it a second time, and I learned quite a bit seeing it the first time. The other two talks have already spawned some changes to plans at PyCon which I will hold off on mentioning until I hear back from people.

Audio Visual

I did not mention this is the volunteer post because I did not want it to get lost there. This year Carl has put together an absolutely insane proposal for AV. He has a dedicated staff for managing the AV complete with capture systems (to capture the raw feed going to the projectors), camera operators, a mixer, and an editor doing real-time production!!! If we wanted to put in some extra effort (and a fair amount of cash) we could broadcast live! I would not expect TED level quality, but the samples produced so far are on par with google talks.

Misc

We gave out a mamoth amount in financial aid (don’t have the number from Ted yet). Open Space use is booming. Sprints are picking up speed and people quite fast. It is going to be a very enjoyable PyCon this year!

Question

The PyCon Lab: Solve This! is most likely going to be held as an Open Space this year. How much interest do people have in this? It has not been properly advertised yet. I am not running it this year, but I will try to poke my head in.

Session Runners

We need Session Runners desperatly. The job is easy; much easier than being a Session Chair (which we also need). You volunteer for a set of 2-3 talks in a session, so it’s not for even a full day. You meet with presenters in the Green Room before thier talk. Help them test their laptop and get thier slides ready. Maybe you get a copy of the slides on a USB drive. You then go with them to the talk and help them hook the machine up if they need help. If there are people interested in an Open Space continuation from the previous talk, you help them get a room and get it posted on the board. If not, help the Session Chair move people out from the provious talk if needed. After that, you do whatever you want until 15min before the next talk. This means that you might miss the beginning or ending of a few talks, but beyond that you get to talk to cool presenters and keep things running smoothly.

The Perks

You get to have a cool radio with headset (either boom, or secret-service style). You get access to Green Room food (just the normal snacks but available durring non-break times). You get to meet and work with the speekers one-on-one. You get to help the conference run much more smoothly, allowing the talk time to be the talk, and not setup or breakdown. You get a ribbon for your badge (if you want it)! By helping out for one session, you are ensuring we have enough volunteers to cover the other sessions so that you are not having talk time wasted watching a presenter set up his laptop instead of presenting. That last one seems like reason enough to me.

A Green Room/OPS for PyCon

Here is the deal. In past years I have noticed a number of small problems which crop up at PyCon, each in and of themselves are not really a big deal. Together they amount to a fair amount of lost time and wasted effort. Much of that wasted effort is in trying to solve specific problems! This year I want to try an experiment to see if we can put in place a framework for dealing with problems in general. The number one way you can improve the ability to deal with a problem at a conference: communication. With that said I will still try to target some specific issues raised in the past.

Session Staff and many department heads will have radios with headsets. We will be able to communicate with key people very quickly and be able to get information out FAST. We could have prevented some rather public issues last year if we had this simple tool alone. These are not toys and we have a limited supply. They are public band FRS radios, so the world will hear us. I won’t bother with more ‘radio’ details here, but the point is between the radios and onside ‘general volunteer mob’ we should be able to get communication to and from key organizers solved.

The Green Room/OPS (behind the registration desk) will be a communication hub with white boards and a fixed radio. Session Chairs whom are stuck in a room will have a means of communicating and getting help at the push of a button. Knowledge of which speakers have arrived already and which are MIA will be known (without having to search stacks of badges). When a speaker, volunteer, or sponsor has a question, they will have a central place from which they can be directed to someone whom can answer thier question. We might not have the answer at OPS, but we will know who does. General attendee questions will be handled at the volunteer center next to registration, or at registration its self.

One of the main things I want to achieve with the Green Room is to give back to the speakers. Those who present at PyCon pay their own way, just like attendees, yet we historically have not treated them very well. These are people whom we are asking to come and present after a grueling review process. They often miss out on the break food. They have to bring their own equipment and set it up them selves (cutting into their presentation time). They are just expected to be where they are supposed to be with no real support or communication. On the other side the Session Chairs and organizers are also volunteers whom pay their own way. They have no clue where the speakers are, what problems are going to come up, and have little forewarning, if any, about special needs of presenters. In the end we have a mass of people at the podium after every talk, no clue if the next speaker is even in the building and the first 5min of every talk is watching the presenter hook up their laptop. In theatre, we call this a farce.

Massive Laundry List O Changes

See.. we do listen

• Public Volunteer Center (next to registration)
• Large board for people to post stuff (‘Hey Bob, we are at Uno’s!’)
• Tons more Lightning talk sessions (two a day!) all via signup sheet at the conference
• Financial Aid forms are dropped off in Volunteer Center (Ted will collect, make checks, and distribute at a set time each day in the green room. Exact details will be sent to those whom received aid via e-mail)
• Session Chairs are provided stop watches
• Session Chairs are provided largeish signs on sticks with 15, 10, 5, and STOP for clearly alerting speakers
• Session Staff are not all alone in a room, but can call on volunteers and will have help for the transitions
• Radios for Session Staff and Dept Heads (mine can make a duck call!)
• Speakers go to the Green Room after registration to check in and get a personalized handout
• Speakers go to the Green Room before talks to get prepped so they are ready to go immediatly when their talk starts
• Speakers and Session Staff are introduced to each other earlier whenever possible
• Printers, a duplicate projection setup, and display adapters in the Green Room will allow problems and talks to be prepped BEFORE the podium
• Speakers and Session Staff have access to snacks in the green room (nothing special, but just this small thing will have a huge impact)
• If presenters or attendees wish to have an Open Space followup, Session Runners will have do the leg work to get the room assigned. Session Runners and Chairs can help prompt people and even make announcements. The Open Space need not occur immediately
• Session Runners will have USB drives for getting the talk slides just before the talk, and get them up on the website for the presenters
• Session Runners and Chairs will be able to edit talk descriptions on the website for adding links and other information promised during Q&A
• Most transitions between talks are now 10min instead of 5! (This will make a huge difference in the talk transitions and overages which always occur)
• Badge Ribbons (for those who want them) The rainbow ‘VOLUNTEER’ ribbon is pretty…. shiny…. you know you want one… you know what you must do to get one…. precious….
• A brief orientation for speakers and session staff on Friday evening to get everyone on the same page. Then anyone whom can not make that session will have plenty of people who can fill them in!
• They key is to get as much information to as many people in a timely manner. If everyone is up to date on the information, then we can work together to solve the problems that arise instead of against each other due to lack of information and co-ordination.

There is a bunch more minor things as well which may or may not occur depending. The Idea is to see what works and what doesn’t, try to only do the things which work, and share that knowledge with everyone else. If something in the above list is causing more problems than helping, then we will stop doing it. Simple as that.

Besides, why travel 30min to see and help a friend when we can both travel half a continent in a few weeks and not talk to each other there!

So here is a public apology of sorts to Ned. I hope we can connect at PyCon and have a chance to chat for more than 5min.

With luck the video of the presentation will be put up on ustream soon like the James Tauber Pinax one was (James is 15min away from me, same story).

1 hour per frame, 24 frames per second, 1 second per day, 40 days.

High res avi is here (14Meg).

We have recieved 109 proposals. (NOTE: there is now a dynamic image URL so you can embed the above graph in your blog! You need to right click as it uses  redirect.)

This number is lower than last year, but that is to be expected. Why? Because:

1. We did not advertise like last year (where we actually paid for professional help to get the call for proposals announced all over the place).
2. We did not announce it in popular IRC channels like in past years
3. We started collecting proposals later than last year
4. We had a much shorter submission period this year (even before last years extension)
5. We did not extend the submission period like we did last year
6. We did not put out regular announcements of the deadline approaching like we did last year
7. There have been a TON of new regional and international python conferences this year
8. The economy?

Seriously, I am shocked that given all that, we still got the number of proposals we did. I am also ecstatic to say that we have 39 reviewers from a broad cross section of the python community. We also have a number of invited speakers. These speakers were selected from the feedback forms from previous years and from general feedback from the greater community. The organizers and the Program Committee reviewed feedback from 08 (the surveys, blog posts, reddit, and mailing lists, IRC logs, etc) in an attempt to discover what things worked last year, the things which did not, and how to do better.

So what is happening now?

The program committee is activly commenting on and reviewing the proposals. At the time of my writting this, there have been 170 comments. We use comments to communicate with the authors for clarifications and to suggest changes. Authors can still edit their proposals and we are working towards reaching a critical mass. As of this post there are only 132 reviews written. That number will grow to well over 400 before we are done. That is a monumantal effort for the program committee, and it is just the tip of the iceburg.

What happens after that?

Once the comment period is over, the PC will will head into the home stretch for reviewing. The proposals will be made read only and the PC will make final review edits, and ensure that there are no proposals which are under reviewed. That is where the fun begins; if you consider trying to get 40 people to all agree on anything fun. I won’t bother rehashing how that happens without thing degrading into chaos except to say that should not given the processes used.

What do you think so far?

There has been quite a bit of discussion on the pycon-pc list on how reviews should be written, how best to communicate issues with the authors via comments, and how to manage the PC effort. Even though the core of the process is stable, every year it is different. One complaint last year was that there was not enough transparency with this process. I have tried to address some of this in blog posts, etc. and this is part of said effort. One missing element is early public discussion on the PC from submitters, reviewers, and the general community. So lets start some of that right now.

If you are so inclined please leave a comment on this post on your thoughts about the process thus far. Please mention if you are a reviewer or proposal author for this year, for a previous year, a past PyCon US attendee, or just someone in the general community.

There are some very strong opinions out there, so please do not get upset just because you don’t agree with someone elses opinion and please keep it civil.

UPDATE: Yes the registration system had issues, we are talking about the program committee. No, authors were not hidden from the reviewers last year, that is a myth I have attempted to debunk time and again. Yes, we all know the sponsor lightning talks were a mistake, that horse is pulp; I will delete any comments brining that up. Same goes for any comments with profanity. *sigh*