Another hiccup

28 Apr

The saga continues. Today while checking up on one of our servers, I noticed that it’s load average was insanely low. It’s one of our “biggest” servers, so it typically deals with a large percentage of our traffic. Load averages over 10 are quite normal, and it was seeing a load average of 2-3.

[email protected]:~# tail /var/log/messages
Apr 28 10:59:56 RHL039 kernel: printk: 9445 messages suppressed.
Apr 28 10:59:56 RHL039 kernel: ip_conntrack: table full, dropping packet.
Apr 28 11:00:01 RHL039 kernel: printk: 9756 messages suppressed.
Apr 28 11:00:01 RHL039 kernel: ip_conntrack: table full, dropping packet.
Apr 28 11:00:06 RHL039 kernel: printk: 6111 messages suppressed.
Apr 28 11:00:06 RHL039 kernel: ip_conntrack: table full, dropping packet.
Apr 28 11:00:11 RHL039 kernel: printk: 3900 messages suppressed.
Apr 28 11:00:11 RHL039 kernel: ip_conntrack: table full, dropping packet.
Apr 28 11:00:16 RHL039 kernel: printk: 2063 messages suppressed.
Apr 28 11:00:16 RHL039 kernel: ip_conntrack: table full, dropping packet.

[email protected]:~# cat /proc/sys/net/ipv4/ip_conntrack_max
[email protected]:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

When the server drops packets like this, things seem randomly and intermittently broken, so we apologize for that. I upped the limit (again) to get things working, but we still don’t know why so many connections are being held open in the first place.


Known unknowns: Grooveshark downtime situation (updated)

23 Apr

Grooveshark is in the middle of some major unplanned downtime right now, after having major server issues just about all day.

What’s going on? We don’t fully know, but we have some clues.

-We are fairly certain that our load balancer is running well over capacity, and we’ve had a new one on order for a little while already.

-Some time this morning, one of our servers broke. Not only did it break, but our alerts that tell us about our servers were also broken due to a misconfiguration (but only for that server).

-Today near peak time, our two biggest servers stopped getting any web traffic except for very short bursts of very tiny amounts of traffic. At the same time, our other servers started seeing insane load averages. /var/log/messages showed warnings about dropping packets because of conntrack limits. The limits were the same as other boxes that had been doing just fine, and in normal circumstances we never come close to those limits, but the counters showed we were definitely hitting those limits and keeping them pegged.

-After upping the limits to even ridiculously higher numbers, the servers started getting web traffic again, and the other servers saw their load averages return to sane values. The connection counts were hovering somewhat close to the new limits, but seemingly with enough margin that they were never hitting the new limits; good enough for a temporary solution.

-Things were seemingly stable for a couple of hours, but connection numbers were still weird. One of our server techs who really is awesome at what he does but shall remain nameless here for his own protection, was investigating the issue and noticed a configuration issue on the load balancer that might have been causing the weird traffic patterns and definitely needed to be fixed. Once it was late enough, we decided to restart the load balancer to apply the settings, a disruption of service that would bring the site down for a minute or so under normal circumstances, well worth it for the chance of having things behave tomorrow.

-Obviously, things did not go quite as planned. The load balancer decided not to come back up, leaving us with the only option of sending a tech out to the data center 1.5 hours away to resurrect the thing.

Update 4/23/2010 11:50am:
Well obviously Grooveshark is back up and running. Total downtime was 2-3 hours, and we have a slightly better picture of what some of the problems were. The load balancer itself wasn’t exactly the problem, it was actually the interaction between the load balancer and the new core switch we installed last weekend. When the load balancer restarted, the switch got confused and essentially got a false alarm on the routing equivalent of an infinite loop, so it cut off the port, making the load balancer completely inaccessible. We now have it set up so that we can still connect to it even if the switch cuts it off, but we also figured out how to work around that issue if it happens again.
There’s still some weird voodoo going on with the servers that we haven’t fully explained, so be prepared for more slowness today while we continue to look into it.


Memcached status like APC status

15 Apr

Just a quick note to point out a cool little utility I discovered recently even though it’s been around for a while:

memcache.php stats like apc.php

If you are at all familiar with the apc status page, you know its nothing fancy but it is a very usable way to gett a quick overview of what is going on, and much prettier than the cli options.

Here is a screenshot from the author’s post:

It literally requires virtually no setup, it’s quite amazing. Optionally change the user/pass combo and add info about your buckets to an array, and boom you’re done. This let’s you easily see how much memory a bucket is using, and you can even see the contents of a slab. It does have one very dangerous feature that I’m tempted to strip out entirely: the ability to flush the contents of a bucket, seemingly with the click of a mouse (I haven’t tested it to find out). Some things should be hard.

Anyway, I highly recommend checking it out, it takes virtually no time and might actually be useful!


Grooveshark Loves Spain

12 Apr

Alternate title: Why is the flag for Spanish the Mexican flag?

As you have probably already noticed, Grooveshark is translated into a couple of different languages now. The language selection dropdown contains the name of the language (in that language), and a tiny icon of a flag. The main purpose of the flag is to be eye-catching, in the hopes that it will be more obvious to non-English speakers that there might be a language that would work better for them. The secondary purpose of the flag, is to indicate the locale. This is where it fails miserably.

The people of Spain are, apparently, quite proud of their country. We frequently receive the complaint that we are Wrong to use the Mexican flag for Spanish. The good people of Argentina, Chile and Colombia don’t seem to mind, but the Spanish, they want to see their flag up there. I guess I can’t really blame them, I mean it is sort of the native home of the language, after all.

The observant will also notice that English shows the American flag, even though English really belongs to the English (that is to say, by that definition, English should have the UK flag next to it). In both cases, the country flags indicate the variant of the language being used. English has the US flag because it’s the US dialect. In the case of Spanish, the translations were done by our very own Carlos Perez whom by now you may have deduced, is from Mexico! Grooveshark uses the Mexican flag for our translation of Spanish because it is written in Mexican Spanish. For that reason, it would be disingenuous for us to use the Spanish flag.

Grooveshark doesn’t have anything against Spain. In fact, Spain is the only Spanish speaking country I have visited, and I found it to be quite nice. Grooveshark has nothing against Britain either, though I have never been there so I can’t speak to the niceness of said country. In any case, as soon as we have the chance, you can bet we’ll be getting the site translated into Spain-Spanish and UK-English, although as you can imagine there are lots and lots of other languages that we need a bit more urgently, such as French, Germain and Russian. Although it might hurt a little bit to see the “wrong” country up there, at least Spaniards can read the site in their native language, aside from the little idiosyncrasies that arise from it being a different dialect.


Attn Hackers: Opening up Grooveshark

21 Dec

At Grooveshark, we get a lot of requests for features that aren’t possible to create in Adobe Flash/Flex/Air or that we just plain don’t have time to do.

In that vein, we’re working on opening up Grooveshark to make it a bit more extensible.

The first thing we’re doing is making the player status really easy to get to. People want their chat clients to be able to reflect what’s playing in the status when they are listening to the desktop app. We can’t do that through AIR directly, and we’re not going to have time to learn about and write special code for each chat client out there, but now anyone who already knows about that stuff or just really wants to put the time into it can do it.
There is a file that we are storing in documentsDirectory\Grooveshark\currentSong.txt
documentsDirectory is defined by Adobe to be the current users’ documents directory. In windows that’s %HOMEPATH%\Documents\Grooveshark\currentSong.txt

The format of the file is pretty obvious if you open it up, but I’ll spell it out here:
where \t is a hard tab (note: names are guaranteed not to have tabs in them)
Valid statuses are “playing,” “paused” and “stopped” – we may add others later if it makes sense to do so. Note that if a user clears their queue or quits Grooveshark, the last song to play will still be listed with a status of “stopped”.

Other ways we plan to open up Grooveshark to 3rd party developers includes creating an external interface through javascript for the web client so people can make Firefox or Chromium extensions to control playback, and creating something similar that can allow the desktop app to accept connections from 3rd party apps running on the desktop alongside, so for example someone could write a plugin that registers global keybaord shortcuts for play/pause, back and next. Again, AIR will not allow us to register those shortcuts directly.


The Case of the Crazy Driver

09 Dec

I often work nights at Grooveshark, which means I also drive home very late or very early, depending on your perspective.

A few times on my way home, I’ve noticed somebody driving in a fashion that can only be summed up as ‘crazy.’ From a distance it looks like they start to pull up into somebody’s driveway, then turn onto the sidewalk, then back onto the road, and then back up onto the sidewalk again, until I or some other car gets near, then they zoom off down some side street.

Tonight I saw that again and I was on my scooter, so I decided to follow the car down the side street. It didn’t take them long to realize they were being followed, so they started driving even more crazily, speeding and running stop signs, turning off on even more side streets, but still sometimes pulling up onto the sidewalk and then veering off. I started to get freaked out at how freaked out this person was, and how crazy they were driving, afraid that they would decide to try to run me over or something, so I gave up. I turned around and took off, but not before looking back. And at that moment the car I had been following was pulled onto the sidewalk again, and I finally saw the detail I had been missing before. What I saw was a newspaper flying out the window of the car. My crazy driver was just a person delivering papers.

No Comments

Posted in life


Bugzilla error

23 Sep

Setting up Bugzilla for a friend on a slicehost account running RHEL5, I ran into issues following the official installation guide.
After running:
./ --check-modules
/usr/bin/perl --all

The error message I received was:

Can’t exec “–decompress”: No such file or directory

Amazingly, I found no exact matches for this in Google (my google-fu is generally weak, however). It turns out that in order for the module installer to be able to install the modules it needs, there’s a certain module it needs!

yum install perl-Compress-Zlib.x86_64
yum install perl-Archive-Tar.noarch

makes everything happy again.

Nothing earth shattering, but maybe now Google will have an answer if anyone else runs into this.


PHP Order of Operations Gotcha

04 Sep

PHP’s decision to give addition, subtraction and string concatenation equal precedence has caused some difficult to track down bugs on several occasions. It’s so non-intuitive I have difficulty remembering this one, and hence keep writing wrong code.


$tacos = "Robots: " . 1 + 2 . " for the win!";

I think a normal human would expect $tacos to be equal to “Robots: 3 for the win!”. But the result is actually “2 for the win!”

What gives? Well, the PHP docs say that plus, minus and string concatenation all get equal precedence, with left associativity. So going from left to right, it says:
“Robots: ” . 1 | “Robots: 1” so far, so good.
“Robots: 1″ + 2 => (int)”Robots: 1” + 2; | “Robots: 1″ converts to 0, so 0 + 2
2 . ” for the win!” | “2 for the win!” D’oh!

I think it would make a lot more sense for string concatenation to take a lower precedence than any arithmetic.

The correct way to write the above code is:

$tacos = "Robots: " . (1 + 2) . " for the win!";


Posted in Coding


Free eBook for VIP users

01 Sep


It looks like there was a mix up and a bunch of our VIP users received the following email:

Andrew Wise
to me

show details 2:10 PM (3 minutes ago)


Thanks for purchasing our eBook, please download the eBook from this email.

Having not actually purchased an eBook, some of our users thought this was a weird form of spam or phishing, or even that their accounts might have been compromised. It’s nothing as sinister as all that, however, just a flub.

Here’s the official explanation from BWC, sent to one of our users who originally pointed out the issue:

Hey [redacted],

Thanks for letting me know, and sorry for the confusion!

Looks like our business guy (Andrew) who handles all our PayPal and such had a little mixup between Grooveshark and one of his side-projects. I just yelled at him, and everything is sorted out now.


Let me know if there’s anything else I can do to help out. Oh, and I guess enjoy being a triplets mommy! ;)


Ben Westermann-Clark


Pizza Hack

23 Aug

Although I know I shouldn’t, I love Stouffer’s french bread pizza.
What I don’t love is the 30 minute cooking time in the oven, or how it comes out all soggy and gross after the recommended 5 minutes in the microwave.

The solution? 1 minute in the microwave, 10 in the toaster oven. The result? Crunchy pizza perfection, in just about 1/3rd of he time.

No Comments

Posted in life