But for some reason I just did not relax. I didn’t get to spend as much time with friends or just vegging on the beach as I wanted, and I really have no one to blame but myself. Part of the problem is that I just could not shut my brain off. Last year I got a notebook, and it ended up being my ‘PyCon’ notebook. This year, I just didn’t seem to have time, oddly enough. Code freeze at work was Friday and that, I am sure, didn’t help. The official fork will most likely be this upcoming Friday.

I have had no time to work on any of my python projects, and it is driving me crazy. There are so many fantastic things happening with django-survey, and Pinax is picking up steam, I need to put the old PyCon ‘08 stuff into archival mode, and start up ‘09. There is some very very interesting stuff going on with the DFW Python group that I want to help out on as well. There are so many fantastic things being worked on right now by incredible people, and I feel a down right claustrophobic not being able to do anything myself. I only been able to attend one Boston Python Meetup so far this year!

It looks like I will be in Montanna the first week of July this year (my anual pre-PyCon-Tech kickoff-kickoff). Barring any project ‘issues‘ I hope to get my act together then (with respect to PyCon software for 2009). If there are any pythonistas in the Missoula area, please send me an e-mail! For now I guess I should get back to unpacking the car.

If Steve is willing to go the full 5K distance, we should be able to support him with some cash. With the exception of this past year, it has been his fund raising efforts which have kept PyCon so cheap. Lets put some of those saved pennies towards a great cause!

At the writing of this there are almost 600 blog posts about the new google hosted application. Most seem to me to be flailing around the actual core of what this new little beastie is. Some are comparing it to Amazon offerings, some as a threat to commodity hosting, and some as the dawn of a new computing revolution. A few highly respected people see this as brining application development to the people the way that html/aol/myspace brought web development to the people. Many see this as a validation that python is an enterprise level platform. While I believe python is just that, I do not yet see this as a validation. The validation comes with Google App Engine’s success. Not that the language needs this added validation. As for the revolution, time will tell.

Google is toting this as making the web a platform. A platform for development, essentially replacing the desktop as where applications get developed and deployed. They do all the busy work of setting up the hardware, configuring systems for monitoring traffic, setting up the database, setting up the source control system, bug tracking, and all the rest, and let you focus on writing the application. Also you get the power of Googles massive data centers with literal warehouses of machines and disks and their custom database.  They bill this as being a platform for building your web based applications, a user base, business, and revenue stream. Of course one revenue stream will be ad-sense further promoting the Google advertising juggernaut. This is all fantastic, but there are limitations (as there must be). I put the limitations at the end.

What is Google really up to?

Google is not releasing the App Engine in a void. There are many other services that google has been rolling out over time (and many quite recently) which need to be looked at in order to get a proper view of what is happening. First lets step back a bit and look at Google’s past. In the past when google released a feature with an API, people would rush out and start building mashups. Mashups which combined parts of Google, and parts of other systems. Systems google often had little or no control over. Early on Google revoked some keys when things got out of hand. Very early on there was some bad PR. Some mashups went away, some came back, some just died out. There was a wealth of data, information, and potential revenue behind those mashups, but it was out of Googles hands by and large.

In the mean time Facebook came along and changed what it meant to write a web app. No longer were applications monolithic disconnected things. They were widgets which plugged into a page. They were cool integrated, socially networked, and shared. They were things people paid real money for. They were things people were using to generate adwords revenue! Google created their own apps. They created them for all the other social sites and the desktop. they did not care who was the hot new trend, as long as they had a share. But they have to play by other peoples rules and API’s.

So google has their search, mail, maps, and documents, online bookmarks, and calendar. They have an rss reader. Others are making mashups, and now many of those are occurring on FaceBook as 3rd party apps. Google releases some extensions for form filling on the docs, and integrating charts. They release a data API. They release OpenSocial as an attempt to standardize all these social networks and the core of what their apps provide; the social connections. They release custom site hosting (without announcing it except in a blog posting). They now have all these great applications and pieces of applications. They have a means of creating, editing, and hosting static html and data. What they lack is a framework to integrate everything. Something where they host the mashups. Something that they can do the deep data mining on. At least that was the case until App Engine came along.

Google claims in the very opening of their announcement that the App Engine is all about the developers. It is all about the people out there who develop neat and interesting things and the feedback loop that creates. The creative creativity of the masses. That has always been the key to Googles overall success. They provide the tools, and others create all those great mashups, sites, and apps. This is not about you creating your cool new app. This is about you creating your cool new Google mashup app utilizing all the other google API’s. They are not all there yet, but they will be. The crucial one, the user backend, is already there. All the other offerings do not require their python API. All the other offerings already have javascript and IFrame, and other means of integrating which were developed for integrating with your blog or MySpace or FaceBook. But make no mistake, they are coming to GAE.

In short this is about taking all those Google pieces parts and creating the ‘next big thing’ and using the developers out on the internet to do it, as they are the ones who will do it anyway and now they can do it for Google.  Google gets their precious data, their add revenue, and at some point people get to pay for the privilege of developing apps for them (either via adds or real money for removing those quotas).

Now comes the really cool part. The SDK includes everything you need for running locally. They have the Google Gears framework for making your apps work both online and offline on your desktop. Integrate all that fully and there really is no difference between your online web based apps and your desktop apps. There is still a long row to hoe before it gets to that point, but the pieces are falling onto place.

Why Python?

There are a number of theories about the real reasons for choosing python. Most believe it’s because python is one of Googles 4 primary languages. I do not believe that exactly. If this could have been done in Java, they would have done that. PHP is the only other ‘language’ that could approach what they want to do. As what they want is a platform for developing mashups with their existing technologies on a massively distributed scale by unknown random people, here is a short list of requirements:

• Easy to develop in (who would develop in prolog?)
• Sandboxable (including no ability to crash the server or corrupt ram)
• No spawning of processes/threads (or other things to bypass cpu/process management)
• No connections in or out of the app except those expressly controlled via an API
• Easy means of administration (for the developers)
• A language for which the Google API’s are already available
• Low overhead for deployment on the servers [initial startup cant be too slow, later requests must be extremely fast]

I know of no other language which meets these requirements. PHP comes close, but would require a partial lobotomy (where python just has some modules removed or limited). Also PHP is not one of the languages that there are API clients for. I know that people are clamoring for other languages. All I can say is don’t hold your breath. I just do not see it happening any time soon.

[UPDATE: as a commenter points out, google is quite dedicated to python and has many core programmers on staff including the language creator. This is a great help for getting things done and adding validity to the project. Read the comments to hear my thoughts on Ruby.]

Growing Pains

App Engine is in its infancy. As with all their Beta projects there are problems. The main problem is how they are dealing with the problems. In short they are overwhelmed. People are asking for PHP, and their favorite python projects to be supported. They made the mistake of claiming that most python frameworks will run on it without putting up the proper CAUTION signs. It is possible to get Zope to run on it with some work. All that is missing is the hook to use the google database instead of the ZODB as the backend, a few minor tweaks, and use the WSGI adapter. Twisted is just out due to the signals, and the threads, and crucially, the tcp connections. One of the problems is that people expect that XYZ module should just work, and it’s the App Engine teams job to do that. The team seems to feel that they provide the framework and others should do the porting. There are also reports of bugs not being responded to in a timely manner. This is a bit laughable given the shear number of bugs currently reported and the 15 or so engineers they have dealing with all the App Engine deployment issues. I am sure that no one expected to have to deal with flamewars in the bug tracker. Or that thousands of people would post +1 comments in the bug tracker making it next to unusable (some people just can’t read instructions). I would not expect all the current bugs to be triaged until late next week or the week after.

Most of the complaints seem to be about the limitations put in place. I can understand that, but I can also understand why hell will freeze over before most of them are lifted. When it comes to an initial deployment it seems quite generous and unrestricted. Insanely so. If you think about what it takes to deploy something like this, at this level, things start to click into place. How would you do it? How would you manage the issues, security risks, vectors of abuse? It is great to say you want to create thread to accept a certified https connection, but if you are making that request, then you have no clue about the technical aspects behind that request or the technical aspects behind the App Engine.

Current Limitations

1. No long running processes

These are run once executions, and there is a time limit of a few seconds. Think of this the same way you would think about a PHP page.

2. No reliable state between runs

There is potential state from one run to the next, but you should not rely on it for large deployments. All state and persistent data should be stored in the database (or via some neat hacks). NOTE: this is more from my reading between the lines and knowledge of load balanced grid deployments. I.e. I do not trust their ‘cache‘ system as something that can be relied upon.Why? Because we are talking about nodes and sandboxes.

3. No incoming TCP connections

No binding to sockets, etc. These are Google’s servers. Even they do not know which node your http request which starts the app will be run on; no way of knowing which IP it really will be. Only apps are running on these nodes. This means no mixing of non-app and app requests. No twisted or zope admin instances. For google to provide a proper balanced network (with proper dispatching), it has to be that way (well at this phase in the game at least).

4. Limited connections out

Google has a url API for making http and https requests out to other servers, a connection to a database and a mail API. Those are the only outgoing network connections, and all are bound in API’s. If you were allowing anyone to run programs on your servers would you want them to be part of botnets?

5. No https

This is not static IP hosting, no cert for you! There are some things that can be done, but there would be cert warnings, etc. Granted this does not stop you from integrating with PayPal, or Google Checkout, where the https checkout is handled by a different site (insanely weak). [UPDATE: yes I know static IP's are not required for certs, but they are required if you would rather people not to get the cert warning or have IE7 mark the site as 'insecure'. And google will not pay for a cert per app, nor will it get a single cert for all apps some of which they are not really sure of their authenticity (a phising app based on Adrian's dynamic html->template tool for instance.) I do expect them to support something in the future, but that is a ways off and will not be for free.]

6.  No spawning new processes (or signal overriding)

Well no big surprise here. Starting new processes could be very dangerous for all, and signal overriding… well that could make it hard for google to safely stop a rogue app (among other things).

7. No creating new threads!

Ok, this is a bit strange at first blush, but if you have ever dealt with grid deployments, or taught a 102 CS course (where you start covering semaphores and mutexes, and IPC) you have had the experience of a rogue multi-thread app taking down a  machine. Part of the problem is that creating a new thread is very much like starting a new process. One of the interesting things about starting a new process is that the operation does not adhere to the nice protocol. It gets the CPU to do that start no matter what, and at the kernel priority (which is not very nice). New threads behave the same way, and are a PITA to deal with when trying to take down a process which has gone rogue. I hated that lab. I have other theories behind why they do not allow this (but that is for another post)

8. No ‘real’ filesystem

Well there is no real access to the file system. Not the ‘real’ file system. As such certain things like tmpfile are not present (as there is no /tmp directory).

9. Crippled import/bytecode

Well that is an overstatement. Google has written their own import replacement, and modified the bytecode (I think) from standard python. I have some theories on why for a later post, but the deal is, forget about using marshal, imp, or even some of the package __import__ hooks, and cPickle is just pickle.  Part of the reason is because of the lack of a ‘real’ file system. The python path and import control is special as only packages from google, and those in your current app are available, and they are specially managed. This should not affect anyone unless you play funky package import tricks that you should not be doing anyway. Extending __path__ in packages does still work, but using __import__ directly to import a package using a computed abs path does not work (might be a bug).

10. Quotas and App shutdown

If your app gets too popular and goes over quotas, then it is disabled. Once it gets too popular,  you need to buy more computes, etc. None of the quotas are set in stone yet, and of course if you use google analytics and/or ad-sense, then the quotas are less restrictive or removed. The details are still in flux. For the beta period you can request a larger quote for free (but each request is reviewed for merit). You can also report app abuse if you find that someone’s app is not being nice.

[UPDATE: Here is the link to the current quota system.]

11. Only 3 Apps and no deleting.

For the beta period, each developer can have just 3 apps, and you can not delete an app.

12. Only pure python

No c extension modules. This is again because of the sandbox system, and all the other stuff above. You can’t prevent process or thread spawning in a c extension. You can not stop a c extension from corrupting things in very bad ways. You can’t stop it from attempting to connect out or bind a tcp port. And it would be a PITA to distribute the binaries to all the nodes like they do for the apps themselves (via custom import hooks + caching).

[UPDATE: fixing numbering and adding some other restrictions and errors people have pointed out]

13.  1MB per file upload limit and 500MB total storage limit.

The 500 MB limit is part of the current Quota system, but I was unaware of the 1MB file upload limit that a commenter pointed out.

14. 1000 files in an app limit

This is a huge problem for people trying to deploy pylons, TG, or Django trunk based applications. One potential solution (which is not currently supported) would be for google to allow for python zip imports and have things bundled.

15. The Google DataStore has limitations over a classic RDBMS

Ben Bangert has a great write up on this, so go read that.

It still will be I think. I will most likely try to rewrite things to give an overview and go into detail on specifics later. Things are getting interesting at work so we will see how much time I have to pull that off.

Files

Ivan beat me to the punch on the main gist post. While at PyCon I had the opportunity to chat with Mike Fletcher, another OLPC volunteer whom I forget their name, Phil Hassey, Richard Jones, Jeff Rush, and about 5 other people who wandered in and out of the small sprint room we were all half passed out in. People came and went durring the discussion (I believe Richard and Phil went off to play a board game at one point as well) which ranged from modern Sci-Fi offerings to games to global warming being a net win for Canada to the history of the world (not the movie). I should have gone to bed well before the discussion started. The discussion turned to the object store on the OLPC platform. Jeff, coming from a ZODB background, was quite pro object store systems replacing ‘file systems’. This is a hot button topic with me. This topic has come up at every professional job I have had going all the way back to when I was an CO-OP at Motorola as a ‘Document Administrator’ (secretary). In fact the only two topics which are more hot button for me are ‘common application UI framework’s, and ‘security after the fact‘. I first started thinking about this subject back in 93 when I first started working on  MUDDs (warcraft, only 100% text for you youngins). The world was editable online (like a lisp MUSH) but also had revision history (via RCS initially). We were dealing with ’serialization’ and how objects were managed. I fell in love with the idea that everything could be described as having a set of attributes (tags) and really you wanted to store and manage these things by those attributes. Permissions were nothing more than attributes. Actions were nothing more than attributes. Meta data by definition were just attributes. We struggled with systems for this, but I came away convinced that we needed a new paradigm in object storage, and this ‘file’ stuff was running on borrowed time. It came up again at Motorola for document management. It came up again at OpenVision (later Veritas) for backup and security compliance. It came up again with ClearCase and Derived Objects. It came up again with ‘dictionaries’ and data management for VoiceXpress. And the code base I currently work on has something called ‘DFiles’ which I can not discuss except to mention the name (DRAT!)

Storage

Back to the discussion at PyCon. I wish I had a transcript of the discussion (no I don’t… I was not as coherent as I think I was). The Idea that everything is just blobs in a cloud of data where the tags determine the meta-structured is nice, but there are some problems. The first and most obvious problem is that it does not integrate well with existing technology and libraries. Decades of software has been written with the concept of files. You can try a fake mapping, and try to integrate things, but it does not work well. Then there is the concept of ’sub-blobs’. That is each of the pieces of data could have sub parts. This maps well to your document which might have a chart or spreadsheet as part of it for instance. This can greatly simplify serialization, and you get all those nice blob store things. Your in-memory structure is your serialization structure. But in reality we already have this. They are called files and directories. It is simply (*cough*) an implementation detail dealing with the storage mapping. Ok, there is nothing simple about it, but we will come back to this. The argument then turned to the fact that you can’t have a blob show up in more than one directory. False. Those are called symlinks, but again that is an implementation detail. One of the biggest benefits of an object-store-as-filesystem is the ability to find and manage things not in a ridged tree structure which does not scale well in the average human brain (where did I put my (ssh) keys again?) But in practice it is just replacing one confusing arbitrary structure with another on some level as it’s usefulness is measured by the quality of the tags, attributes, and indelibility of the data.If you had those things well defined in a directory tree structure, then it works just as well (as google desktop search proves). A more subtile problem is that not all tags/attributes are created equal. It took a long time for my betters and practical experience to prove this to me. Many attributes are only useful to programs. These programatic tags are for relating data, validation, encoding, and the like. Most of the time these are auto generated or involve mathematical computations. They are never intended for human interpretation,but are none the less crucial for data management. You can try to predetermine the different types of these meta attributes, or just lump them together, but neither of these approaches are really tractable. Spend some time deep diving into the abuses of the windows registry and you begin to get an idea of the issues.

I know I am glossing over all the details, and not really giving any points the attention they deserve. I am not even properly quantifying the points. Issues of language are completly being skipped over (try describing what a ‘word’ is in your application; try again when that application deals with speech and natural language… how does that abstract into meaningful tags?) Oh well. The point is there must be a happy median. We should be able to have something which has a file system programatic interface, as well as a generic data store interface. The browsing of the data should be an abstraction. If this is implemented with a classic journaling file system or in a database should be an implementation detail at the filesystem level. Why invent a new abstraction layer which everyone must now implement against when we have a perfectly good one that everyone already does? A file by any other name still contains data. If this is such a good model, think about extending it to namespaces. The problems in software code management (which is just data on a very real level) for which namespaces were invented exist on the filesystem as well. Chew on that while you code with Matrix.Optimizer and Optimizer.Matrix.

Google

Google has an interesting take on all of this. All of their service (news, documents, reader, calendar, mail, blogger, etc) all have a file like data storage for the objects represented. They use folders/directories (really tags). The only restriction is that the folders are only one level deep. I do not care for this myself. I would love to be able to have a ‘people’ folder under my ‘python’ folder and have only those times tagged with both ‘python’ and ‘people’ under that ‘folder’. Maybe that is just me. I would not want these sub folder relations to be automatic. I would want control over the layout, but have the population automatic. But that is the only extension to their system I would like to see. Beyond that it just works. It works with both the object store model and the file/directory model. If only google would open up their API’s a bit more to include this system. On wait, they just did. You know if I had hit ‘publish’ on this post last evening when I first wrote most of this, I would have been ‘prophetic’ or at least ‘first post!’.

It’s not all hearts and ponies and sparkle (even if it is python and an abstraction layer on top of django to boot!!!) I have been holding off on posting this err… post until I could formulate a non-reactionary opinion on the entire Google Apps thing. I now have an opinion and it is much along the same lines as Duncan McGreggor. The issues I have are both similar and yet unique to his, and I will post on them separately.

Python, and Conference Software

This post is already too long,and my laptop battery is dying (no the charger is at work ). Those of you that I talked with at Pycon about UnConference hosting know what this is all about, and I told you so . The last piece just fell into place. With that, good night

I had a wonderful talk with Michael Trier and Brian Rosner who do the ‘This Week in Django‘ podcast. Lots of useful information, check it out. (And you get to hear my voice….)

UPDATE: Here is the image I mentioned in the podcast about [name withheld] watching a presentation Guido gave at google, on google video, at PyCon 2007, while Guido was giving a talk. Thanks again to Sean of Tummy.com for doing the networking last year, and making so that people could do things like this and not have it effect the network!

We have a new registration system this year. It has its quirks, and there have been some issues. Most of these were expected and we had plans in place to deal with them. AMK has taken on the role of the Registration Manager. With things in his more than capable hands everything is running quite smoothly. I can’t help but stick my nose back in to resolve some issues. To be honest I do not have the temperament (or time) to deal with talking to actual people (code is more my thing). But some people I already know, or requests were made and I knew the person was still online. AMK didn’t help things by documenting some standard responses to some of the common problems, which made it too easy for me to ‘help out’ . As of this writing we have more registrations than we had last year at the close of the early-bird period (375 in 07, and 390 right now).

So why do I have the Blues? I can’t decide which tutorials to take. Work is paying for me to go this year (and has even ponied up for a sponsorship!) That means that they have a say. This is additionally complicated by the fact that I still don’t have my XO, and it’s questionable that I will get it in time. What this means is I am waffling on whether to take ‘SWIG Master Class (David Beazley)’ or ‘Making Small Software for Small People, Sugar/OLPC Coding by Example (Mike C. Fletcher)’. I really want to take the Sugar class. We use SWIG at work, and not the simple stuff either. We don’t do any of the C++ stuff, but have things like type safety on our constants with custom repr’s and the ability to do things like pass in an array of length 0 and say its of length 100; have to test those memory error conditions after all. Callback functions are supported, oh and it’s thread safe, with all calls releasing the GIL safely. SWIG gets really painful at that level.

In the afternoon it looks like ‘Tools for Scientific Computing in Python (Travis Oliphant and Eric Jones)’ for me. There were hard choices here, I want to learn WxPython, and the generator tricks look very interesting, but in the end SciPy is the best choice both for work and for fun. With my existing experience I feel safe in skipping the morning companion tutorial. We use the SciPy packages including numpy and matplotlib quite a bit at work but my experience with the later is only cursory. I hate to admit it, but many times I use win32all to push the data into excel and generate graphs that way. I have it on my list to write an FFT and a group theory approach to pitch detection using SciPy; yes that is ‘fun’.

The evening is the toughest. This is where PyCon-Tech related tutorials come to the fore. It is down to ‘Practical Applications of Agile (Web) Testing Tools (C. Titus Brown and Grig Gheorghiu)’ and ‘Django Code Lab (Jacob Kaplan-Moss, Adrian Holovaty and James Bennett)’. I would love to go to the code lab and dive into parts of PyCon-Tech. The hard part would be selecting something small enough to discuss. PyCon-Tech has grown quite large and some of the parts are quite involved. Just giving an overview of the registration system can take 3 hours (from actual experience). There are parts of the proposal system which need optimizing (but again explaining the issues alone could take too long). The real problem I would love help with is the separation of the display (html templates) from the core of what I now think of as the PyCon-Tech framework. We have a new design, but once again the design is coupled with the implementation. Swapping out another design is easier than last year, but harder than it was just two months ago. Then there is the 10K lb. Gorilla in the corner. PyCon-Tech has 0 automated testing. Guess I will have to go for the testing tools, and hope to snare the Django folks some other time. To be honest I feel weird with the Idea of bringing PyCon-Tech stuff to a tutorial; as if I am stealing time from attendees for conference related stuff.

At some point I need to do a post on the registration system… The idea was to keep it simple. Just a minor update to the old cgi form with the data stored in a database instead of a text file. Three hour chat to give an overview…. For the masochistic, you can read the comments in the source code here and here (which are incomplete) and some other notes here and here (a little out of date).

Within 5min I had broken out Wing IDE and was coding franticly. It was a shear joy to think about. The API is simple, and robust, and I knew exactly the chart I wanted to make. Not a simple dinky chart, but a full fledged stacked area line chart with color for multiple data plots and a ton of data for each plot. One of the additions to the proposal system is a complete change history on the proposals. This is implemented with the django admin log, so that even changes made in the admin are captured (though with less detail). You can perform one of three actions; add, change, and delete. We have four primary objects in the proposal system; proposals, reviews, comments, and attachments. New attachments are considered proposal edits for the graph, and you can not edit or remove comments or attachments. You can not delete anything. So this broke down into 5 data plots; new proposal, edit proposal, new review, edit review, comment. I wanted to show each of these changes on a per day basis, with the space accounting for the total number of edits having occurred to that point in time. In short I dove into one of the harder graphs. In truth the real work was generating the data I wanted to plot. The Google Chart API is a generic plotting api, so you need to scale and convert your data to match the type of graph you want to plot. Not a big deal, and anyone who has done any real work with plotting packages can do this blindfolded, and the Google API is so simple it makes it fun.

With that done, I decided to change my code a bit to add the ability to restrict the time frame, so I could see the post-deadline changes only:

The code to generate the graphs is now part of the project, and I will integrate it somehow for next year. In total I spent about an hour, and it was quite an enjoyable hour. With that done, I showed off my work to a few people, and then went back to the blogs to see what people were saying about the API.

What I read shocked me. There were a number of people who really liked the API, but I could not find anyone who had actually bothered to fully read the documentation or use it! There were complaints about negative numbers and hidden or undocumented features, and other garbage that is not really worth discussing. There were a few people who ‘got it’, and in general people thought it was cool and interesting. There were a few rants that are not worth mentioning.

So I think I will tackle the biggest misconception I have seen thus far. The Google Charting API is NOT for plotting your raw data points! It does not deal with dates. It does not deal with scales. It does not deal with negative numbers, log scales, or fancy data; but it CAN plot them! Why? Because it is just a basic plot package, and it has to deal with the restrictions that are placed on URL’s, as that is the data transmission layer. This is what I mean. If you have say data which ranges from 0 to 10, and you send Google that data to plot. It will plot it all right, but it will only plot it in the bottom 10% of the graph. You need to scale the data up to one of three plot ranges. The docs go out of their way to highlight the distinction between actual and plotted data. These ranges are determined due to the limitations of the URL. The first one is the ‘simple‘ 0-60 scale, supported by the simple encoding. this is the best way to get a lot of data points sent to google. This is because it only requires a single character per data point. If you don’t mind the simple resolution of only 61 discernible points on the Y axis, this is for you. The second is the ‘text‘ encoding; a 0-1000 range using essentially the percentile notation with one decimal point. The docs say 0.0 -> 100.0, but 1K by any other name is still 1K. As this requires a whopping 5 characters per data point average (have to count the separator), I see no reason to every use this, but I understand that it is good for people who want to be able to read and hand type the chart data. The last is the ‘extended‘ base64 2digit character encode which allows for a range from 0-4095. Depending on the encoding you use, you must rescale your data to match. So if you have negative values in your data, you must shift and rescale your data, and draw a new X axis line on the chart where your scaled 0 value is. Why not have the API support math conversions? Because there are so many, and because of the URL encoding. You have a limited number of characters in a URL, and you need to optimize for that.

Lets dive deeper into the ‘extended‘ encoding. This is most likely the only encoding I will ever use. Unfortunately the javascript sample code it comes with is complete garbage, and at first blush looks broken, limiting people to only 3844 unique values. I fear it will turn people off to what is a very simple encoding. Lets walk through an evolution of encoders in python [NOTE: all code on this page is in the public domain]:

GC_EXTENDED_MAP = (
    'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    'abcdefghijklmnopqrstuvwxyz'
    '0123456789-.')
gc_extended = lambda num: GC_EXTENDED_MAP[num/64]+GC_EXTENDED_MAP[num%64]

Here we have a very simple number to 2 digit google extended encoding. This assumes that the number has already been scaled to between 0 and 4095. Yup thats it folks. So now what we want is to have something deal with the scaling for us.

GC_EXTENDED_MAP = (
    'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    'abcdefghijklmnopqrstuvwxyz'
    '0123456789-.')

def gc_extended(num, max=4095):
    scaled = (num*4095)/max
    return CG_EXTENDED_MAP[scaled/64]+CG_EXTENDED_MAP[scaled%64]

this is better, but there are still some issues. The first one I will tackle is the ’rounding’ issue. Now the sample javascript code the google provides uses round. I personally do not like javascript’s round implementation as I have found it to be very slow on IE 6. I have no clue why. I also rarely use round in python when dealing with integer numbers. This is mainly due to laziness, but more on that in a bit. Unless you have a raw data range of around 6K or greater, you do NOT need to use round. This is because you are scaling up, and the difference of 1 is most likely going to be well within your delta error, or below your percision anyway. But lets say you do want to deal with rounding issues, and we can deal with floating point data at the same time:

    scaled = int(((num*4095.0)/max)+0.5)

Done. Cheap and sleazy round and no ‘import math’ needed. this is also faster in javascript on IE 6; MUCH faster (no clue why, maybe their round is a wrapper for fround?) Ok, back to some real issues. Many times you will want to plot negative numbers. You could shift your data, and then scale. or you could manage that as part of the encoding. Managing it as part of the encoding is slower in some respect (as you are repeating math), but it does simplify the code a bit.

def gc_extended(num, min=0, max=4095):
    scaled = ((num-min)*4095)/(max-min)
    return CG_EXTENDED_MAP[scaled/64]+CG_EXTENDED_MAP[scaled%64]

Now we are getting some where. But there is one last issue. We may not know the full range of values, but instead want to Ceil/Floor errant values. For example, when dealing with signal processing, I know the signal will normally be within a given range (+-10db), but I also know that sometimes plugging in or unplugging equipment can cause measurement spikes in the data at 100db. These values are ‘real’ in that they happened, but would skew the graph. We want to peg those to the local min/max:

def gc_extended(num, min=0, max=4095, floor=0, ceil=4095):
    if num < floor: num = floor
    if num > ceil: num = ceil
    scaled = ((num-min)*4095)/(max-min)
    return CG_EXTENDED_MAP[scaled/64]+CG_EXTENDED_MAP[scaled%64]

There, that simplifies things, and with floor and ceil as pre scaled values, the api is simpler (if a bit slower). One last thing to deal with. Many times you need to deal with missing data points. Say you are plotting two lines which data collected at different points on the X axis. You have three options. You can give google the data for both X and Y in pairs, or you can give the data for just Y, and fill in the missing points with the ‘missing’ data marker ‘__’ (the simple and text encodings also support missing data), or a combination of both. This can drastically reduce the length of the URL encoding the data if the union set of missing unique x values is large and you go with supplying pairs, or small and you go with additional missing markers.

GC_EXTENDED_MAP = (
    'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    'abcdefghijklmnopqrstuvwxyz'
    '0123456789-.')

def gc_extended(num, min=0, max=4095, floor=0, ceil=4095, missing=None):
    if num == missing: return '__'
    if num < min: num = floor
    if num > max: num = ceil
    scaled = ((num-min)*4095)/(max-min)
    return CG_EXTENDED_MAP[scaled/64]+CG_EXTENDED_MAP[scaled%64]

There; done. That is all the python code for dealing with encoding any range of numbers onto a scaled google chart data extended encoded number. You can have your ‘missing’ data point be -1 and it will still work just fine. You can Ceil and Floor, and deal with all the rest. If you use the django curry utility, you can have even MORE fun! Lets take a look:

def encode_data(raw_data):
    """encode the data for plotting 5 standard deviations from the average
       computed with missing data treated as 0.0 for avg compute
       (common bell curve compute)
    """
    dev = standard_deviation(raw_data) * 5
    avg = sum(x for x in raw_data if x != -1)/len(raw_data)
    encoder = curry(gc_encode, min=avg-dev, max=avg+dev, missing=-1)
    return ''.join(encoder(num) for num in raw_data)

I love python. Now lets bring this full circle back to that blog post Marty made on data visualization in Django. With not too much difficulty we could construct some standard data based graphs for the django DataBrowse contrib app. These would be simple, but extensible graphs. There would be a django view for generating the Google Chart API url, and would then return a redirect to that URL! No charting packages or javascript to write. Just some simple python, which unlike other charting packages can be checked into the Django contrib, as it does not require any other packages. Another cool use is having dynamic up to date charts in your S5 presentation! This is just to0 cool.

This does not negate the need for proper cache use, but it can be a life saver when you want to update your sites templates, but don’t want to restart the server because people are say, using it to submit proposals before a deadline. This app is not feature complete. Currently only the following cache backends are supported:

• dummy
• simple
• localmem

The remaining backends need to be implemented. The database and file based backends are simple to do, but memcached will take a bit more work. The system is extensible, so you can extend it to work with your own cache backends the same way you can with django’s cache system. All you have to do is extend the base class the same way you would with the django cache system:

class BaseCacheMgr(object):
    """No description has been set.
    """
    has_cull = False
    scheme = 'unknown'

    def __init__(self, cache, host, params):
        self.cache = cache
    def __iter__(self):
        """Should be overridden to iterate over the cache and return
        a dict with the form {'key': key, 'short_key': short_key,
                                     'repr': cache_value,
                                     'expires': datetime_expires, 'expired': False}
        """
        raise NotImplementedError
    def info(self):
        default = (str(self.cache.default_timeout)
                   if hasattr(self.cache, 'default_timeout') else 'None')
        return [{'name': 'Scheme', 'value': self.scheme},
                {'name': 'Description', 'value': self.__doc__},
                {'name': 'Default Timeout', 'value': default},
                {'name': 'Has Cull', 'value': repr(self.has_cull)}]
    def clear(self):
        raise NotImplementedError
    def cull(self):
        if not self.has_cull: return
        raise NotImplementedError
    def delete(self, key):
        self.cache.delete(key)

Fairly self explanatory when you look at the image above. Take a look at the simple backend implementation for more details.

Other features that need to be implemented:

• Clear Expired Button
• Paged View (like the admin)
• Sort Table heading links (like the admin)
• Search Fields (like the admin)
• Move the repr/short_key/expires helper code into the base class.

One of the stated goals of PyCon-Tech is to give back to the python community which makes the conference possible. This series is my attempt to shine a light on some of the general use applications under the hood and how you can use them for other projects. The first app in this series is the feedutil, a lightweight generic RSS/Atom feed pull. (it is also the only app in PyCon-Tech being used for other projects that I know of).

Overview

Feedutil is a lightweight app for pulling RSS/Atom feeds onto your site with django template tags, or custom views. This is not a full feed aggregator like feedjack, but you could write one with it. This is more along the lines of the blogger plugin which allows you to have the latest 5 entries from an RSS feed appear on your sidebar. We use it on the PyCon website for the main about page which has summaries of the latest PyCon Blog entries via Atom, and on an organizer page which replicates a Trac RSS issue feed for open website bugs. This does not use any django models, and there is no database interaction. You could create your own models for managing your feeds, but that is not the purpose of feedutil.

The feedutil provides two primary template tags {% feed feed_url [posts_to_show] [cache_expires] %} and {% get_feed feed_url [posts_to_show] [cache_expires] as var %}. There is also a higher level interface to feedparser which includes caching pull_feed(feed_url, posts_to_show=None, cache_expires=None) => posts_dict.

Requirements

Before we go any further, there are some requirements for feedutil:

• Python2.5 (sorry, I like typing ‘res = x if test else y’)
• django 0.96 or greater
• a properly configured django cache backend (not really needed, but highly recommended)
• feedparser (feedparser.py needs to be on your python path)

Settings

There are some configuration settings you can set in your settings.py (plah). They all have reasonable? defaults:

• FEEDUTIL_NUM_POSTS (default: -1)

Max number of entries to pull from the feed. Use -1 to pull all entries.

• FEEDUTIL_CACHE_MIN (default: 30)

Number of minutes to hold the feed in cache before polling the feed again. this is to set the default cache limit. each feed can have their cache time/poll frequency set independently. Use 0 to never cache. Cache backend must be set for non-sero value.

• FEEDUTIL_SUMMARY_LEN (default: 150)

Each post gets a html-stripped text ’summary’, and this is character limit on those summaries. A ‘…’ will be appended if the summary does not fit in this limit. This uses the django template filter striptags to remove the html tags before applying the character limit.

• FEEDUTIL_SUMMARY_HTML_WORDS (default: 25)

Each post gets a ’summary_html’ entry which preserves the html tags, but limits the number of words in the rendered html output. this uses the django template filter truncatewords_html to get the work done.

The {% feed %} tag

{% feed feed_url [posts_to_show] [cache_expires] %}

{% feed “http://www.dougma.com/feed/atom/” 1 30 %}

• feed_url – literal string or variable which resolves to a valid feed url (required)
• posts_to_show – number of entries you want shown (default: FEEDUTIL_NUM_POSTS)
• cache_expires – #min before the cache expires. (default: FEEDUTIL_CACHE_MIN)

{% load feedutil %}

{% feed "http://www.dougma.com/feed/atom/" %}

Yup, that is all you need to reproduce a feed on a website. Its minimal, but it works. The default html template used to render this tag will look something like the following per feed entry:

<p class="feed-post">

<h2><a href="http://www.dougma.com/archives/47" title="HELP!">HELP!</a></h2>

<cite>doug</cite>

... post content omitted....

<h3>Published: Thu, 20 Sep 2007 08:57:05 -0500</h3>

This is using a django inclusion tag, and the default template ‘feedutil/feed.html’. You can override this to make a new custom default, or you can pull in the feed as a veriable into your django template and have per-feed customization (next section). There are optional arguments to the {% feed %} tag, ‘posts_to_show’, and ‘cache_expires’. These have the obvious effect.

The {% get_feed %} tag

{% get_feed feed_url [posts_to_show] [cache_expires] as var_name %}

{% get_feed “http://www.dougma.com/feed/atom/” 5 30 as posts %}

• feed_url – literal string or variable which resolves to a valid feed url (required)
• posts_to_show – number of entries you want shown (default: FEEDUTIL_NUM_POSTS)
• cache_expires – #min before the cache expires. (default: FEEDUTIL_CACHE_MIN)
• var_name – name of the variable to assign the post list to (required)

The variable will be assigned a list of dictionaries, each entry in the feed being a dict with the following keys:

• title – the post title text.
• url – the url for the entry.
• author – the author (if available)
• published – datetime object for the publish date
• summary -Text summary generated from the post content with tags removed, and limited to FEEDUTIL_SUMMARY_LEN characters. (see settings above)
• summary_html – The content html limited to FEEDUTIL_SUMMARY_HTML_WORDS words. (see settings above)
• content – the content of the entry.
• comments – Sometimes a link to the comments section, sometimes the number of comments as a string, sometimes the actual html for comments. and sometimes something weird. All depends on what is generating the feed., and whether or not it conforms to the standards.

With this you can create any custom html you need for your feed reproduction. Let’s look at the template code for the actual pycon 2008 website.

{% load feedutil %}{% load appmedia %}
{% get_feed "http://pycon.blogspot.com/feeds/posts/default" 5 as news %}

<div id="lfcolumn">
<h3>Latest News
    (<a href="http://pycon.blogspot.com/">PyCon Blog</a>)
    <a href="http://pycon.blogspot.com/feeds/posts/default">
        <img src="{% app_media_prefix 'website' %}img/rss.png"
                  alt="Subscribe to the RSS Feed" />
    </a>
</h3>

<ul id="feed_entries">
    {% for entry in news %}
        <li> <a href="{{ entry.url }}" title="{{ entry.title|escape }}">
            {{ entry.title|escape }}</a>
        <br/> {{ entry.published|date }}
        <br/> <em>{{entry.summary}}</em>
        </li>
    {% endfor %}
</ul>
</div>

The pull_feed() function

pull_feed(feed_url, posts_to_show=None, cache_expires=None) => list of entry dicts

This is the worker function behind feedutil. You can use this in your django python code to grab the feed as a list of python dictionaries with a common interface across RSS, RSS2, and Atom feeds. You have the added benefits of the summary, summary_html, and caching. Simple, elegant, and gets the job done. This function is most often used in custom views where you want to merge the data from multiple fields and sort them by the ‘published’ datetime object. This is a simple way to create a feed aggregator.

def aggregator_view(request, myfeeds):
    """myfeeds must ba a dictionary of {'title': 'http://feed/url/'}"""
    feeds =  [pull_feed(url) for url in myfeeds.itervalues()]
    entries = sorted(chain(*feeds), key="published", reversed=True)
    return render_to_response('feeds.html', { 'feeds': myfeeds, 'entries': entries},
            context_instance=RequestContext(request))

There, a feed aggregator in 5 sloc, not counting imports and actual feed data, and well the html

So I will be taking part in the sprint. I have no axes to grind, so I will most likely focus on bug reports. I live in western Mass, and work in the Burlington area, so my ‘Boston’ location is a bit disingenuous. I know there are a number of Django people in the Mass area, and I would love to meet up somewhere to coordinate. My current plan is to commute into Burlington and camp out at Borders at WaySide Commons (google map). I picked this location for many reasons, but am not committed. (Reasons include: away from home, coffee, free internet, coffee, public transportation, coffee, across the street from work, coffee.) So if anyone is interested in meeting up, let me know!