RSS
 

500,000 Users and scaling

29 May

Grooveshark surpassed the 500,000 registered user mark today.
Ignoring the fact that many of our users never bother to register (it’s not necessary in order to use the site), 500k is an absolutely phenomenal number, especially compared to where we were just a year ago: 33k. The scary thing is that under our current growth rate, we will have over a million registered users in roughly 3 months.

Can we double our capacity in just 3 months? Obviously, history implies that it’s possible; we’ve already done much more than that. In fact we’ve done better than that: with little change in infrastructure and much of the same server capacity we’ve managed to make Grooveshark faster and scale at the same time.

On the other hand, much of the low-hanging scalability fruit has been picked now. We use memcached extensively, use a master/slave DB configuration with a data warehouse for logging or writes that don’t need to be processed in real time, and have begun doing some rudimentary sharding for stream-related activities.

What’s left? Well, we aren’t yet at the point where we can scale linearly simply by adding more servers, except probably for streaming servers. For that we need more sharding, primarily. There are still some SQL optimizations that can be made, like bringing session ids down to 16 bytes from 32 (32 on disk and 96 in memory, thanks to utf8) and ultimately getting them out of the database altogether, and using memcached even more heavily, but really all of those things only buy us time. Not that there is anything wrong with buying time, because we also need time to work on new features like last.fm scrobbling, a super-secret redesign, launching on half a dozen mobile platforms, etc., all with a relatively small dev team, but ultimately there are some fundamental architecture changes coming, and if we’re going to keep doubling our number of users every 3 months, it’s going to have to be very soon.

Update: A lot of people have been looking at this post as evidence that we are working on scrobbling support. I should point out that scrobbling support now exists for VIP users: Enable it here.

 
 

Leave a Reply

 

 
  1. Tyler Francis

    July 15, 2009 at 10:49 pm

    Woot for groveshark, and double woot for scrobbling support!

    I might be alone in this, but I find it very hard to listen to music that doesn’t scrobble. As Hype Machine put it, “If it doesn’t scrobble, did I really listen to it?”

    Keep up the good work!

     
  2. teo

    November 14, 2009 at 2:47 am

    last.fm scrobbling please ;)

     
  3. Eric

    December 23, 2009 at 10:49 am

    So how is the last.fm scrobble support coming?

     
  4. Jay

    December 23, 2009 at 11:18 am

    Well let’s put it this way, I just scrobbled a song from Grooveshark yesterday. ;) It pretty much works but the UI needs improvement.

     
  5. Dimitar Panov

    February 14, 2010 at 8:34 am

    Any news on scrobbling support? :-)

     
  6. Jay

    February 14, 2010 at 10:17 am

    There is, but not much that I can talk about. Let’s just say the code is written but is waiting on something very special so it can be released…

     
  7. Danny

    February 17, 2010 at 6:40 pm

    Can’t wait till scobbling works w/ groove shark! anyupdates??

    Great work!

     
  8. Chad Johnson

    May 8, 2010 at 12:24 pm

    Good information on scaling.

     
  9. Last Year « Jay Paroline – Grooveshark Dev

    May 19, 2010 at 8:22 pm

    […] was almost a year ago that I made a post about hitting the 500,000 user mark and talked about some of our anticipated scaling […]

     
  10. Brian Kenyon

    July 28, 2010 at 4:33 pm

    Yes I love the scrobbling available in the VIP account however, many of us are still waiting for scrobbling support from the Android mobile app as well and not only the website.

     
  11. Jay

    July 28, 2010 at 11:36 pm

    I agree, but the mobile teams have been very busy making their apps not suck. :) They’ve seen quite a bit of spit and polish and are getting to the point where they’re pretty stable now and almost feature complete, so I hope we will start to see features like scrobbling make their way in. There’s still a few things I’d like to see them do first though, like give me access to my library and my favorite playlists.