Jay Paroline – Grooveshark Dev

When it comes to disks, raid and otherwise, there’s a lot of confusion about how to optimize for different server configurations, and unfortunately much more emphasis seems to be placed on sustained throughput, with little mention going to IOPS or concurrency.

Here are grooveshark we use Sun’s X4500, aka “Thumper” for housing your MP3s. Lately we’ve been hitting some buffering issues during peak hours, where it seemed like the 4500 was not able to keep up with the requests coming in. Although we are growing rapidly, we should still be nowhere near saturating the IOPS of 48 drives, but iostat -Cx was showing that the drives dedicated to serving up content had transactions waiting 100% of the time, and the disks were busy 100% of the time. Service time was in the triple digits. Insanity. Something was obviously misconfigured.

We were using 3 zpools for our content, each configured in raidz with about 8 drives (if I remember correctly). Ok, so we actually only have the IOPS of 24 drives in that configuration, but we still should not be anywhere near saturating that. After a fair amount of digging, I discovered that raidz is probably the worst configuration possible for serving up thousands of 3-5mb files concurrently. That’s because raidz causes every drive to engage in every read request, no matter how small the file. That means your theoretical transfer rate for any given file is N times the transfer rate of each drive, but your IOPS is the same as one drive, no matter how many drives you have in your pool. In other words, adding more drives to a raidz increases disk transfer bandwidth, but does nothing to alleviate the overhead associated with random access seeks. This configuration is ideal if you are transferring multi-gigabyte files frequently, but at 3.5-5MB, seek time makes up almost all of the overhead, and it would be much better to have N drives all seeking to N files simultaneously.

I brought up these concerns to our sysadmins, and they set up a small mirrored pool to handle our most popular content, and the performance difference is quite astounding even with just two disks in that pool. In a mirrored configuration like this, each drive can respond to requests for data separately because they each have the full set of data. Adding 2 more drives to the 24 drives already in production nearly doubled our IOPS capacity, because we went from the IOPS of 3 drives to the IOPS of 5, nearly doubling our capacity with only two hard drives. Our system admins will be adding more disks to the mirror to give us even more breathing room. We’ll have to wait and see tomorrow if the added IOPS capacity eliminates the buffering issues that users have been running into lately, but I bet it will.

I’m still getting used to not calling it “Grooveshark Lite.” For those who haven’t noticed, it’s been re-branded as just plain old Grooveshark. On one hand, it’s an awesome testament to just how successful the project has been. On the other hand, it’s a little harder to talk about it now. I can take credit for a large portion of Grooveshark Lite, but I can’t really take credit for a large portion of Grooveshark; Grooveshark represents so much more to me than just our player.

Anyway, thanks to a nifty little Perl script that Travis helpfully rewrote for me to be more compatible with our hosting environment, I was able to grab art for an extra 57,000 or so albums that we had not been able to get from our partners previously. What does this mean for you, the user? You should see less of this:

in your queue and song info panels, and more real album art, like this:

What’s the magic trick? Simple, the script looks for art embedded inside the mp3s that we have associated with an album, guesses at which piece of art is best. It’s surprising how many mp3s actually have art embedded in them.

Hopefully this will make Grooveshark more usable. I know I find it very frustrating when I have a large queue filled mostly with the default album art; it’s very hard to tell where you are when everything looks the same.

Google released their Chrome Browser today. I’ll refrain from giving it a full review as many others have already manged to do so.

I want to comment on the ease of adding search engines and some minor tweaks you can do to make the search functionality more powerful, specifically in regards to Grooveshark Lite and TinySong.

To add a search engine to Chrome, just visit the site. If they have properly implemented their search extensions by embedding information about them in the page (and we have), then all you have to do is visit the site and Google will add it to its list of search engines. Each search engine has a “keyword” that you type before your search term to indicate which search engine you wish to use. The default keyword is the domain of the search engine, but you can change it by clicking on the wrench icon, and then Options | Basics | Default Search | Manage

I changed mine so that I can search lite by typing in “listen artistname” so if I want to listen to Soltero I type “listen Soltero.”
Likewise I modified the keyword for TinySong to be ts. So if I want to search for Camera Obscura, I type “ts Camera Obscura”

Another thing about Google Chrome that is really cool as it relates to Grooveshark Lite is the built in support for making any website an application. I installed Grooveshark Lite on my start menu, and now it runs as an independent application with the lite Favicon showing up in the taskbar. Very cool.