Archive for April, 2008


24 Apr

I’m back from my mini-vacation and it sounds like things are crazy at the office, but I think even so that I should have time to start writing again and I look forward to doing so.

My first order of business tonight is to start reading Systemantics: How Systems Work and Especially How They Fail by John Gall which was recommended by Raymond Chen and therefore automatically must be worth reading. Because I’m sure many of you are too lazy to actually follow that link, here’s the relevant RChen review:

Systemantics is very much like The Mythical Man-Month, but with a lot more attitude. The most important lessons I learned are a reinterpretation of Le Chatelier’s Principle for complex systems (“Every complex system resists its proper functioning”) and the Fundamental Failure-Mode Theorem (“Every complex system is operating in an error mode”).

You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed.

That’s why I’m skeptical of people who look at some catastrophic failure of a complex system and say, “Wow, the odds of this happening are astronomical. Five different safety systems had to fail simultaneously!” What they don’t realize is that one or two of those systems are failing all the time, and it’s up to the other three systems to prevent the failure from turning into a disaster. You never see a news story that says “A gas refinery did not explode today because simultaneous failures in the first, second, fourth, and fifth safety systems did not lead to a disaster thanks to a correctly-functioning third system.” The role of the failure and the savior may change over time, until eventually all of the systems choose to have a bad day all on the same day, and something goes boom.

Time to hit the books!


Grooveshark Lite Credits

16 Apr

Grooveshark Lite is up and the backend seems to be relatively stable now. I’ve been playing music seamlessly for a couple of hours. I noticed that we don’t have any specific information about the GSLite Team, so I thought I’d put up something quick:



John Ashenden


Katy Richard

Katy is our flex/flash guru. Many perceive her powers to be magical, despite her protestations to the contrary.


Jay Paroline (me!)

I created a SOAP interface using NuSOAP to allow Katy’s front end to communicate with the API that I wrote, which does a combination of DB work and communicating with the existing Grooveshark framework.

Streaming media server

Qiu Yuanzong, Paulo Dasilva, Edwin Fuquen

None of these guys appear on the about page right now, so I can’t include their pictures. They got Red5 up and running on 3 amazon ec2 servers in preparation for launch.


Colin Hostert

Colin set up the ec2 service, put up a bunch of front-end web nodes to load balance across, and made sure we had access to memcached.

Oh shit, stuff is broken

Travis Whitton, Chris Suter, Douglas Bell

When Red5 started crashing hardcore on our ec2 servers once hundreds of people tried to use it at the same time while the red5 team was out, Travis and Chris jumped in to debug the problem and hack together some temporary solutions. Meanwhile, Douglas got us hooked up with a few more ec2 instances to help balance the load.


Amazon S3 and EC2
Cloud computing FTW! While long-term this is an expensive solution, when you need instant scalability, that’s what these services are for. Once we have time we’ll have in-house solutions for both hosting cached MP3s (currently on S3) and streaming them (currently on EC2).
MySQL (InnoDB)
All of our data exists in a massive MySQL database. It took about 6 hours worth of UPDATEs and ALTERs to the schema to get the DB in shape for Grooveshark Lite.
SOAP is a bulky way to communicate, and we ran into limitations both in NuSOAP, PHP SOAP and Flash in terms of implementations, but we got it working anyway. In the next iteration of GS Lite we will most likely forgo SOAP in favor of JSON-RPC or some other lightweight protocol.
The highest levels of navigation all reside in memcached, which give us the massive benefit of removing a huge amount of load from the MySQL DB. I was also able to decrease the load on our web servers (building XML in an interpreted language is very expensive!) by adding some hacks to NuSOAP so that any message that contains non-volatile information is cached after it is built. I believe that memcached deserves the bulk of the credit for why the back end was able to withstand the massive amounts of load we were under even when red5 couldn’t handle the strain.
Red5 is an open source clone of flash media server written in Java. Despite the problems we’ve had with it, without Red5, GS Lite would not be possible.
All of our in-house servers are Sun boxes.
Backend programming language of choice for Grooveshark. It may not be as sexy as Ruby or Python, but it’s hellaciously powerful.
Goold ol’ Apache, what would we do without you?
Flex is an incredibly powerful open source framework for creating complex flash apps easily. From what I gather from Katy, Flex’s default behavior is both a blessing and a curse; often making things pretty by default, but a horrible pain to override if you need it to do something different.


It goes without saying that GS Lite would not be at all possible without the existing Grooveshark infrastructure, so really the whole Grooveshark development team deserves credit.


Meet Grooveshark Lite

15 Apr

Well, for better or worse, Grooveshark Lite is up live and running.

Things are a bit buggy because someone thought it would be a great idea to exponentially increase the complexity of our web setup at the exact same time, and it’s already generated a lot of attention, so don’t be surprised if things just stop working intermittently.
All that said, Grooveshark Lite makes listening to music online super easy, you should definitely check it out.

Here’s a more formal description of GS Lite:

What Grooveshark Lite is:
A flash program written with flex that lets you listen to any MP3s we have in our system.
You can search for music or you can navigate through the heirarchy to find music, browsing songs, artists, albums, playlists or genres.
You can play any of the music you find by clicking the play button, the add to queue button, or by dragging it to the queue.
You can save your queue as a playlist.
You can browse your own music and playlists (even if it’s on another computer) by going to My Music.

What Grooveshark Lite is not:
Grooveshark Lite does not have any social networking components, and will not.

What Grooveshark Lite isn’t yet but will be soon:
Some features that didn’t make it into the initial release (tambourine), but will be added very soon:
Playlist editing: We have a slick interface for managing your playlists; in addition to creating playlists you’ll be able to add & remove songs, delete playlists, rename them, add descriptions, etc.
Auto-DJ: build a queue and our recommendation will add new songs to the queue that it thinks you will like.
Share your music: When you make a playlist, you can get embed code to put our player on your site. There’s a tiny version you can stick in a sidebar or a less compact version to fit in a blog post. The widget pulls the songs based on a playlist, so if you want to change the songs on your site, for example, all you have to do is edit the playlist.
Share your music part 2: If you don’t want to embed the player or don’t have somewhere to do so, you can get a link to your playlist that takes your friends to a page with the playlist loaded in the player. For example, my cake playlist will be accessible via: http://{gslite URL}/boxmonkey/cake or you can send your friends to http://{gslite URL}/{username} and they can browse all of your playlists.

Apologies if any of this is incoherent: I’ve been up since 1pm yesterday. AWESOME.


What’s been keeping me busy

14 Apr

Katy and I have been hard at work this past month on a top-secret project, code-named tambourine. It will be known to the public as Grooveshark Lite.

We’re nearing the end and we’ve been given permission to leak a tiny little bit. There’s other big news coming down the pipe very soon, but this is pretty big to us too. Teaser screenshots:

Grooveshark Lite:


Listen to any song cached on Grooveshark:

Unlike the current Grooveshark website, Grooveshark Lite doesn’t require the sharkbyte client, and doesn’t require a user account to listen, search or browse.