What I really want to talk about is an example of why it’s a good idea to make wrappers for PHP extensions instead of just using them directly.

When Grooveshark started using memcached ever-so-long-ago, with the memcache pecl extension, we decided to create a GMemcache class which extends memcache. Our main reason for doing this was to add some convenience (like having the constructor register all the servers) and to add some features that the extension was missing (like key prefixes). We recently decided that it’s time to move from the stagnant memcache extension to the pecl memcached extension, which is based on libmemcached, which supports many nifty features we’ve been longing for, such as:

• Binary protocol
• Timeouts in milliseconds, not seconds
• getByKey
• CAS
• Efficient consistent hashing
• Buffered writes
• Asyncronous I/O

Normally such a transition would be a nightmare. Our codebase talks to memcached in a million different places. But since we’ve been using a wrapper from day 1, I was able to make a new version of GMemcache with the same interface as the old one, that extends memcached. It handles all the minor differences between how the two work, so all the thousands of other lines in the app that talk to memcached do not have to change. That made the conversion a <1 day project, when it probably would have otherwise been a month long project. It also has the advantage that if we decide for some reason to go back to using pecl memcache, we only have to revert one file.

Example:

$tacos = "Robots: " . 1 + 2 . " for the win!";

I think a normal human would expect $tacos to be equal to “Robots: 3 for the win!”. But the result is actually “2 for the win!”

What gives? Well, the PHP docs say that plus, minus and string concatenation all get equal precedence, with left associativity. So going from left to right, it says:
“Robots: ” . 1 | “Robots: 1″ so far, so good.
“Robots: 1″ + 2 => (int)”Robots: 1″ + 2; | “Robots: 1″ converts to 0, so 0 + 2
2 . ” for the win!” | “2 for the win!” D’oh!

I think it would make a lot more sense for string concatenation to take a lower precedence than any arithmetic.

The correct way to write the above code is:

$tacos = "Robots: " . (1 + 2) . " for the win!";

As Grooveshark evolves and the application becomes more sophisticated, the PHP layer is more and more becoming just an interface to the DB. The application just needs the data; it knows exactly what to do with it from there. It also only needs to ask for one type of data at a time, whereas a traditional webpage would need to load dozens of different pieces of information at the same time. So for our type of application, magic methods and ORM can really just get in the way when all we really need is to run a query, fill up an array with the results of that query, and return it.

Our old libraries employing ORM, magic methods and collections, were designed to meet the needs of a typical website and don’t necessarily make sense for a full-fledged application. On a webpage, you might only show 20 results at a time, so the overhead of having a bunch of getters and setters automatically fire whenever you load up your data is probably not noticeable. But in an application, you often load far more results than can be displayed, and allow the user to interact with them more richly. When you’re loading 500, or 5000 results as opposed to 20, the overhead of ORM and magic can start to really bog you down. I first noticed the overhead issue when testing new method calls for lite2, when in some cases fetching the data would take over 30 seconds, triggering my locally defined maximum execution time, even when the data was already cached.

Like any responsible developer considering making changes to code for performance reasons, I profiled our collections code using XDebug and KCachegrind, and then I rewrote the code to bypass collections, magic and all that stuff, loading data from the DB (or memcache) into an array and returning it. The difference? In the worst case, bypassing magic was an order of magnitude less work, often times far better than that. My > 30 second example took less than 1 second in the new code.

For Grooveshark 2.0 code, wherever makes sense, we are bypassing magic, ORM and collections and loading data directly. This of course means that Grooveshark is faster, but it also means that we can load more data at once. In most cases we can now afford to load up entire lists of songs without having to paginate the results, which in turn means fewer calls to the backend *and* much less work for the database. Whenever you must LIMIT results, you must also ORDER BY the results so they come back in an order that makes sense. Not having to ORDER results means in many cases we save an expensive filesort which often requires a temporary table in MySQL. Returning the full data set also allows the client to do more with the data, like decide how the results should actually be sorted and displayed to the user. But that’s another post…

The solution? Add a way to detect crawlers, and don’t give them sessions. Most of the solutions I’ve seen online look something like this:

function crawlerDetect($USER_AGENT)
{
$crawlers = array(
array('Google', 'Google'),
array('msnbot', 'MSN'),
array('Rambler', 'Rambler'),
array('Yahoo', 'Yahoo'),
array('AbachoBOT', 'AbachoBOT'),
array('accoona', 'Accoona'),
array('AcoiRobot', 'AcoiRobot'),
array('ASPSeek', 'ASPSeek'),
array('CrocCrawler', 'CrocCrawler'),
array('Dumbot', 'Dumbot'),
array('FAST-WebCrawler', 'FAST-WebCrawler'),
array('GeonaBot', 'GeonaBot'),
array('Gigabot', 'Gigabot'),
array('Lycos', 'Lycos spider'),
array('MSRBOT', 'MSRBOT'),
array('Scooter', 'Altavista robot'),
array('AltaVista', 'Altavista robot'),
array('IDBot', 'ID-Search Bot'),
array('eStyle', 'eStyle Bot'),
array('Scrubby', 'Scrubby robot')
);
foreach ($crawler as $c) {
if (stristr($USER_AGENT, $c[0])) {
return($c[1]);
}
}
return false;
}

Essentially, doing a for loop over the entire list of possible clients, and searching the user agent string for each one, one at a time. This seems way too slow and inefficient for something that is going to have to run on essentially every call on a high volume website, so I rewrote it to look like this:
public static function getIsCrawler($userAgent)
{
$crawlers = 'Google|msnbot|Rambler|Yahoo|AbachoBOT|accoona|' .
'AcioRobot|ASPSeek|CocoCrawler|Dumbot|FAST-WebCrawler|' .
'GeonaBot|Gigabot|Lycos|MSRBOT|Scooter|AltaVista|IDBot|eStyle|Scrubby';
$isCrawler = (preg_match("/$crawlers/", $userAgent) > 0);
return $isCrawler;
}

In my not-very-scientific testing, running on my local box my version takes 11 seconds to do 1 million comparisons, whereas looping through an array of crawlers to do 1 million comparisons takes 70 seconds. So there you have it, using a single regex for string matching rather than looping over an array can be 7 times faster. I suspect, but have not tested, that the performance gap gets bigger the more strings you are testing against.

For example, let’s say we have a page that shows a user their playlists if they are logged in, but this page also has other useful information on it, and users aren’t required to log in. In our original design for our authentication class, if authentication “failed” (i.e. the user was not logged in), a call to Auth::getUserID() would return an empty string.

With that implementation, every piece that might have something to display to a logged in user has to have a bunch of conditional logic checking to see if the user is logged in, doing one thing i they are and another if they are not, adding unnecessary complexity and, of course, more potential for bugs to crop up.

This type of logic is completely unnecessary. I changed Auth to return a userID of 0 if the user is not logged in, and now it is not necessary to have any special handling for that case. If the user is logged in, they get playlists. If the user is not logged in, userID 0 does not have any playlists so they do not get playlists. I estimate that this simple change made over 100 lines of code obsolete. Not a whole lot of code in the grand scheme of things, but how many bugs can hide in 100 lines of code?

Another example of the same principle involves exceptions (which, as regular readers of Raymond Chen know, are hard to deal with). Under certain circumstances, recommendations from an external source can be included with our normal set of recommendations. The author of the recommendation-supplementing code originally had it throwing exceptions whenever it had problems. Of course, the case of not having supplemental recommendations because of an error is not really a special case. As far as my code is concerned, you just don’t have supplemental recommendations. In this case modifying the original code eliminates unnecessary handling for a special case and eliminates the potential for an uncaught exception to slip through.

A general rule of thumb to help avoid unnecessary special cases, at least with PHP, is to always return what you say you are going to return (and don’t throw exceptions). If your method processes some data which results in an array, return an array even if the processing has no results. This is the way most native methods in PHP already work, if you think about it. count(array()) doesn’t return an empty string or null or raise an exception, nor does count(null). Really, neither of these are special cases. In either case, the number of elements is zero, and developers are not required to care about the differences.

We thought it was a permutation of this bug which is fixed, but apparently this is a new one, so we are going to file it.

I find it a bit disturbing that the related bug was brought to their attention in 2006 and the limited fix was only released to the general public 2 months ago. Over two years to release a fix to a language bug?!? From the comments on the bug, they do not seem to take it very seriously:

Fixed in the avmplus mainline. I’d suggest a 10.0 target date, since the change will need soaking time, there is a work-around and it is rare to encounter this problem.

Change 243008 by rreitmai@rickr-dev on 2006/09/08 12:37:23

Severity: important
Summary: Verifier error bugfix

Detailed Description:

An optimization in the verifier allows us to avoid checking for null in various circumstances. Unfortunately, we were being a little too aggressive and we missed a case. If a block is a target of a backwards branch then we need to assume upon entry of the block that no check has been performed.

The other way to fix this would be to emit null checks just prior to the branch for any values that have notNull true, but this could also create a bunch of unnecessary checks.

Aren’t obscure language bugs the absolute worst kind to have? In my case, it first went undetected because it didn’t trigger any errors/warnings with the debug version of flash (although our sample condensed code does), and then it was extremely difficult to track down because THE CODE WAS CORRECT, and when the production version of flash fails, it does so silently.

First I ran into a horrible language bug that causes a nasty runtime crash wherein Flash complains that it can’t resolve a class with itself. I wasn’t doing any weird voodoo with class definitions and Katy, our primary Flex developer, couldn’t find anything wrong with my code, but rewrote some of it anyway. Poof, problem fixed.

Everything looked fine so we thought we were ready to launch. So we did last night. Skyler complained that album art wasn’t loading for him, but he had just reinstalled 64 bit flash on 64 bit Linux, and he’s had weird problems with that before. We were unable to reproduce, attributed it to Flash+Linux weirdness and ignored the issue. Today, more people complained about it. We were not able to reproduce. Eventually I figured out that the bug was only affecting non-debug versions of Flash, but I couldn’t imagine what would be causing album art to not display: my code has nothing to do with displaying images, and I could see in Firebug that the art was being fetched properly, nothing was wrong with the headers, etc.. But it definitely worked fine in the debug version of Flash. Katy looked at it, and through process of elimination was able to determine that the problem was somewhere in my code, but couldn’t tell where exactly. Everything looked valid, and from the perspective of the debug version of Flash, it was, but my code wasn’t very “actionscripty” – because I’m used to writing PHP and haven’t had any sort of training on actionscript beyond looking at existing code and occasionally looking at a language reference book. Katy couldn’t figure out what was wrong with my code, so she rewrote it, and now everything is back to normal again.

I don’t yet know what part of my code was causing the problem, so on Monday I plan to reinstate my code, downgrade to the non-debug version of flash, and figure out what the heck is causing the bug. My instinct tells me that it’s probably a combination of an optimization setting in non-debug Flash combined with my use of Actionscript in a way that is technically valid but that no Actionscript coder would ever normally use it. For example, Katy says I was using a generic Object as a Dictionary, because I was actually trying to replicate associative arrays in PHP, I didn’t know about the Dictionary object, and when I had asked previously how to do that, I was told to use an Object. :P

The good news is that I don’t think many people outside the company paid much attention to this bug. They were too busy being excited about the new features that Katy introduced: broadcast to twitter, broadcast to facebook, and deep linking support: When you “byte” a song (open the info panel), your URL bar changes. You can simply copy the URL to share with a friend, *and* clicking back and forward work, so if you want to go from the song you opened back to the playlist you found it on, just click back in your browser. Freaky, but it works.

Abstractions do not really simplify our lives as much as they were meant to. [...] all abstractions leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don’t save us time learning [...] and all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder

In my case, I was not working with programming tools per se, more of an abstraction of an abstraction built into the Grooveshark framework that is meant to make life as a programmer easier. And normally it does. But in this case, that abstraction was wrapped in a couple more layers of abstraction away from where my code needed the information, and somewhere in there the information was, for lack of a better description, being mangled. The particular form of mangling is actually due to a lower level abstraction at the DB layer, but is not handled in the higher level of abstraction because for most uses it doesn’t matter.

From the level of abstraction where my code was sitting, the information needed in order to un-mangle the data was simply not available. I went through the various layers to find a convenient place to put the un-mangling, but by the time I found a place, everything was so abstract that I couldn’t confidently modify that code and see all potential ramifications, so I just did an end-run around many of the layers of abstraction. This is certainly not ideal, but it works. The point is, an abstraction intended to make life as a programmer slightly easier most of the time, can easily make life as a programmer significantly more difficult on edge cases. Unfortunately, once a code base has been established, most cases are edge cases: feature additions, new products built on top of the old infrastructure, etc., all or most unforeseen, and therefore unaccounted for during the original design process.

The first was when we were using a tool to test the scaling performance of a new feature. According to this tool the performance was absolutely abysmal with only 50 virtual clients, and I spent a couple of days writing alternative algorithms and trying to test them with this tool. The scalability was horrible no matter what I did, so I added some logging to see if something was going on that wasn’t supposed to be. It was. The testing tool was sending completely wrong information, creating completely unrealistic scenarios. The testing implementation was done by another developer who was sure that he had set it up right, and I was not familiar with the tool so I assumed that it was a problem with my code. Having another developer write the tests was supposed to save me time.

The latest monitoring snag was caused by a monitoring tool that, for our convenience, tails the php error log and sends the last couple hundred lines to us whenever there is an error. We kept getting the same error periodically and could not track down the source; everything seemed to be working perfectly, but we still got the error. Usage of the site every day is breaking new records so we assumed that it had something to do with the unprecedented load on servers, but could never pinpoint the source of the errors. Today I looked a bit more closely at the error log only to discover that the log was from 3 days ago!

So the question is, when your testing or monitoring tools report an error, how do you ensure that the error is real and not just in the tool itself?

It’s interesting, then, that I work at Grooveshark where much of that dream is being fulfilled by participating in this movement. The one piece that is missing is having a channel to share music with other people and subsequently helping small bands by making them more discoverable. Well, now with the release of Autoplay in Grooveshark Lite, it’s kind of like I get to be everybody’s DJ. Of course a computer scientist would write a DJing program rather than doing the manual labor of DJing.

As Professor Fishman, the best professor who ever lived, was fond of saying in our classes, a computer scientist isn’t satisfied with just using computers to put other people out of a job, they won’t settle until they manage to put themselves out of a job too. To be fair, he usually talked about that in the context of AI and specifically programming languages such as LISP, where the program can rewrite itself, but I think it applies here as well.

Now I get to be everyone’s DJ, but with everyone’s help too. If the system is currently a bad DJ, keep giving it feedback and it will learn. Imagine if you got to call up your local radio station and yell at them every time they played something you didn’t like, and congratulate them every time they played something you liked. If they didn’t block your phone number, you’d end up with the ultimate radio station for you, and that’s what Grooveshark aims to be, although we admit it will take some time to get there.

Check out Autoplay, and let me know what you think.