Jay Paroline

We have a some PHP files that run as cron jobs. Some of them run frequently, but occasionally take longer to run than the interval we have them set to. To prevent pile-up, I created a simple lockfile mechanism. I take advantage of posix kill 0 to check if the script is still running, so I think this is only guaranteed to work on linux, definitely does not work on Windows last I checked.

All you need are two functions:

function lockfile_open()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        $pid = file_get_contents($pidfile);
        $running = posix_kill($pid, 0);
        if ($running) {
            return false;
        } else {
            unlink($pidfile);
        }
    }
    $handle = fopen($pidfile, 'x');
    if (!$handle) {
        return false;
    }
    $pid = getmypid();
    fwrite($handle, $pid);
    fclose($handle);
    register_shutdown_function('lockfile_close');
    return true;
}

function lockfile_close()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        unlink($pidfile);
    }
}

To use it, simply call lockfile_open(); at the beginning of cron execution and check the return, if it returns false then you have another copy of your script already running and you need to handle appropriately (in our case we log an error and die).

As with my previous post, I’ll break this down line by line to explain what is happening.

$pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';

this sets the path for the lockfile/pidfile. I am storing them in /tmp and basing the name entirely on the name of the script that is executing. Depending on your system you may need to store them somewhere else and come up with a different naming scheme.

if (file_exists($pidfile)) {
    $pid = file_get_contents($pidfile);

we check to see if our pidfile already exists. If no other copy of the script is running, it should not exist because the shutdown handler deletes the file. If it does exist then we need to read the contents of the file to get the pid, so we can check to see if it’s really still running, or if it’s stale.

$running = posix_kill($pid, 0);

doing a posix kill with signal 0 does not actually send a signal to the process, but still runs error checking, which includes testing to see if the pid is running.

    if ($running) {
        return false;
    } else {
        unlink($pidfile);
    }
}

if posix_kill returned something truthy then the process is in fact still running, so we return false to indicate that. If the pidfile is actually stale, we delete it and then continue as if the file never existed.

$handle = fopen($pidfile, 'x');

We create the pidfile with mode x, this will fail if the pidfile already exists, which means this handles the possible race condition of two scripts executing at the same time, where both check above for the pidfile and both see it as not existing. mode x guarantees that one of those scripts will succeed in creating the file and the other will fail.

if (!$handle) {
    return false;
}

If we failed to create the pidfile then we assume it’s due to the aforementioned race condition, so return false. NOTE: this assumes permissions are set up correctly for wherever you are saving your pidfile, if you set up a path that your crons can’t write to, this will fail to create the pidfile and assume it was due to the race condition even though there is probably not another instance of your script running.

$pid = getmypid();
fwrite($handle, $pid);
fclose($handle);

Here we get our current pid and write to to the pidfile, then close the file.

    register_shutdown_function('lockfile_close');
    return true;
}

If we made it this far then we are the first/only instance of the script currently running. We register our lockfile_close() function to clean up our pidfile when the script stops executing, and we return true to indicate that it’s safe to proceed.

function lockfile_close()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        unlink($pidfile);
    }
}

lockfile_close gets called automatically by the shutdown handler when the script stops executing. We check to make sure the pidfile still exists and then delete it, so we aren’t leaving stale pidfiles around.

South Carolina’s confederate flag has come under intense scrutiny due to the recent actions of a certain racist asshole. Apparently last year, Governor Nikki Haley defended the confederate flag saying that CEOs do not have a problem with it.

I have two observations to make about this:

1. If a CEO is talking to the governor about moving their business to SC or has already moved their business to SC, they most likely had already decided the confederate flag was a non-issue for them, so why would they bring it up to the governor? The fact that CEOs aren’t talking to her about it doesn’t seem to say a whole lot about how CEOs in general feel about it.

2. Even if CEOs have no problems with the confederate flag, I know that SC is losing talent because of it. I know this because I know 4 extremely talented designers and software engineers from SC who left because of the blatant state sanctioned racism including the confederate flag. I also know that the confederate flag keeps talent away because although SC has the ideal climate for me and meets my requirements of being coastal, I refused to consider any jobs in that area because of the confederate flag. I will not tolerate living anywhere that is OK with prominently displaying a blatant symbol of racism. So even if CEOs are not complaining about the flag, it’s bad for business because it’s driving talent away, and businesses need talented people.

Now, I know some people claim that the confederate flag is not a symbol of racism and it’s about heritage, or it has some other meaning I’m not familiar with (I was not raised in the south, so I admit I may be ignorant on these things). Let’s assume that they are right and for some people the confederate flag has nothing at all to do with racism. I would argue that it does not matter what the flag means to some, if to the majority of people it represents racist ideology. Take the swastika: it had different meanings to different cultures, but you don’t see non-racists flying swastika flags and defending it as the ancient buddhist symbol of eternity, do you? No. Since the confederate flag is so closely associated with racism, it’s time for the non-racists who use it to mean something about heritage (or whatever they think it means) to pick a new symbol that does not have the hateful overtones of the confederate flag, unless they really don’t mind being associated with white supremacy…in which case I would argue that they are more racist than they think they are.

Discovered this amazing XSS cheat sheet while trying to prove to a co-worker that using a regex to prevent <script> tags embedded in HTML was not going to be effective. I knew about the UTF8 vulnerabilities and all of the obvious ones, but the US-ASCII Encoding one especially was new to me, impressive!

Grooveshark is looking for talented web developers. For part 3, I’m listing our front end developer/designer (affectionately known here as “devsigner”) position.

Grooveshark Web Developer / Designers

Must be willing to relocate to Gainesville, FL and legally work in the US. Relocation assistance is available.
Responsibilities include:
Maintaining existing HTML templates and JavaScript code
Writing clean well-formed HTML and CSS
Writing JavaScript to be able to achieve the desired user experience
Using Photoshop and Illustrator and be able to convert images to markup and JavaScript
Optimizing HTML and CSS for speed considerations

Desired Qualities:
Enjoy writing high quality, easy to read, well-documented code
High level of detail given to converting designs
Able to follow coding standards
Good written and verbal communication skills
Well versed in best practices & security concerns for web development
Ability to work independently and on teams, with little guidance
Capable of prototyping features (either visually or concretely)
Capable of having a flexible schedule

Experience:
Experience with JavaScript, HTML & CSS
Experience in Photoshop and designing websites
Familiar with jQuery, including JavaScriptMVC
Familiar with Actionscript
Experience with version control software

Not Desired:
Uses WYSIWYG editors
Only searches for jQuery plugin that fits the job

As you may already know, Grooveshark’s Android app was recently <a href=”http://news.cnet.com/8301-31001_3-20051156-261.htm”>pulled from the Android market</a> due to either label pressure, or concerns from its competing music service that is soon to be launched (or both). That event has been covered in great detail all over the net, so I won’t rehash it here; that’s not what this blog post is about.

Yesterday we discovered that Grooveshark has also been pulled from the Chrome Web Store. Fortunately, if you already installed Grooveshark, Google does not appear to be revoking the app from users Chrome home pages.

For users who had not yet installed Grooveshark, they are out of luck from the web store. However, we have discovered that it is possible to “jail break” your Chrome installation to get the full Grooveshark + Chrome experience.

Step 1: Visit http://grooveshark.com in your chrome browser.

Step 2: Close the Grooveshark tab.

Step 3: Open the New Tab tab by typing CTRL+T (CMD+T in OSX?), or by pressing on the small + icon next to the rightmost tab.

Step 4: If Grooveshark appears under “Most visited,” simply drag it to the top-left position, hover over the site preview for a second, and click on the pin icon that appears. Skip to step 7.

Step 5: If Grooveshark does not appear under Most visited, scroll down to Recently closed, and click on Grooveshark.

Step 6: Proceed to step 1.

Step 7: Collapse the Apps section, or remove it entirely.

It may take several thousand iterations of this loop to bring Grooveshark into your Most visited section depending on how obsessively you are checking the 8 websites that appear there for you, but we feel this added effort on your part is well worth it. When you are done Chrome should look something like this:

P.S.: Grooveshark was ranked #8, right behind YouTube, an interesting juxtaposition since these services are both powered by user contributed content, and are both legal, but one is owned by Google and the other is not.

P.P.S.: For those of you who are really bad at detecting sarcasm, I am being facetious. As with every blog post I make here, I also am not speaking for Grooveshark in any sort of official capacity.

At Grooveshark, Gearman is an integral part of our backend technology stack.

30 Second Introduction to Gearman

• Gearman is a simple, fast job queuing server.
• Gearman is an anagram for manager. Gearman does not do any work itself, it just distributes jobs to workers.
• Clients, workers and servers can all be on different boxes.
• Jobs can be synchronous or asynchronous.
• Jobs can have priorities.

To learn more about gearman, visit gearman.org, and peruse the presentations. I got started with this intro (or one much like it), but there may be better presentations available now, I haven’t checked.
The rest of this article will assume that you understand the basics Gearman including the terminology, and are looking for some use cases involving a real live high traffic website.

Architecture

With some exceptions, our architecture with respect to Gearman is a bit unconventional. In a typical deployment, you would have a large set of Apache + PHP servers (at Grooveshark we call these “front end nodes” or FENs for short) communicating with a smaller set of Gearman job servers, and a set of workers that are each connected to all of the job servers. In our setup, we have a gearman job server running on each FEN, and jobs are submitted over localhost. That’s because most of the jobs we submit are asynchronous, and we want the latency to be as low as possible so the FENs can fire off a job and get back to handling the user’s request. Then we have workers running on other boxes which connect to the gearman servers on the FENs and process the jobs. Where the workers run depends on their purpose, for example workers that insert data into a data store usually live on the same box as the data store, which again cuts down on network latency. This architecture means that in general, each FEN is isolated from the rest of the FENs, and Gearman servers are not another potential point of failure or even slowdowns. The only way a Gearman server is unavailable is if the FEN itself is out of commission. The only way a Gearman server is running slow is if the whole FEN is running slow.

Rate Limiting

One of the things that is really neat about this gearman architecture, especially when used asynchronously, is that jobs that need to happen eventually but not necessarily immediately can be easily rate limited by simply controlling the number of workers that are running. For example, we recently migrated Playlists from MySQL to MongoDB. Because many playlists have been abandoned over the years, we didn’t want to just blindly import all playlists into mongo. Instead we import them from MySQL as they are accessed. Once the data is in MongoDB, it is no longer needed in MySQL, so we would like to be able to delete that data to free up some memory. Deleting that data is by no means an urgent task, and we know that deletes of this nature cannot run in parallel; running more than one at a time just results in extra lock contention.

Our solution is to insert a job into the gearman queue to delete a given playlist from MySQL. We then have a single worker connecting to all of the FENs asking for playlist deletion jobs and then running the deletes one at a time from the MySQL server. Not surprisingly, when we flipped the switch deletion jobs came in much faster than they could be processed; at the peak we had a backlog of 800,000 deletion jobs waiting to be processed, and it took us about 2.5 weeks to get that number down to zero. During that time we had no DB hiccups, and server load was kept low.

Data Logging

We have certain high volume data that must be logged, such as song plays for accounting purposes, and searches performed so we can make our search algorithm better. We need to be able to log this high volume data in real time, without affecting the responsiveness of the site. Logging jobs are submitted asynchronously to Gearman over localhost. On our hadoop cluster, we have multiple workers per FEN collecting and processing jobs as quickly as possible. Each worker only connects to one FEN — in fact, each FEN has about 12 workers just for processing logging jobs. For a more in depth explanation for why we went with this setup, see lessons learned.

Backend API

We have some disjointed systems written in various languages that need to be able to interface with our core library, which is written in PHP. We considered making a simple API over HTTP much like the one that powers the website, but decided that it was silly to pay the cost of all the overhead of HTTP for an internal API. Instead, a set of PHP workers handle the incoming messages from these systems and respond accordingly. This also provides some level of rate limiting or control over how parallelized we want the processing to be. If a well meaning junior developer writes a some crazy piece of software with 2048 processes all trying to look up song information at once, we can rest assured that the database won’t actually be swamped with that much concurrency, because at most it will be limited to the number of workers that we have running.

Lessons Learned/Caveats

No technology is perfect, especially a technology when you are using it in a way other than how it was intended to be used.
We found that gearman workers (at least the pecl gearman extension’s implementation) connect to and process jobs on gearman servers in a round-robin fashion, draining all jobs from one server before moving to the next. That creates a few different headaches for us:

• If one server has a large backlog of jobs and workers get to it first, they will process those jobs exclusively until they are all done, leaving the other servers to end up with a huge backlog
• If one server is unreachable, workers will wait however long the timeout is configured for every time they run through the round-robin list. Even if the timeout is as low as 1 second, that is 1 second out of 20 that the worker cannot be processing any jobs. In a high volume logging situation, those jobs can add up quickly
• Gearman doesn’t give memory that was used for long queues back to the OS when it’s done with it. It will reuse this memory, but if your normal gearman memory needs are 60MB and an epic backlog caused by these interactions leads it to use 2GB of memory, you won’t get that memory back until Gearman is restarted.

Our solution to these issues is, unless there is a strong need to rate limit the work, just configure a separate worker for each FEN so if one FEN is having weird issues, it won’t affect the others.
Our architecture combined with the fact that each request from a user will go to a different FEN means that we can’t take advantage of one really cool gearman feature: unique jobs. Unique jobs means that we could fire asynchronous jobs to prefetch data we know the client is going to ask for, and if the client asks for it before it is ready, we could have a synchronous request hook into the same job, waiting for the response.
Talking to a Gearman server over localhost is not the fastest thing in the world. We considered using Gearman to handle geolocation lookups by IP address so we can provide localized concert recommendations, since those jobs could be asynchronous, but we found that submitting an asynchronous job to Gearman was an order of magnitude slower than doing the lookup directly with the geoip PHP extension once we compiled it with mmap support. Gearman was still insanely fast, but this goes to show that not everything is better served being processed through Gearman.

Wish List

From reading the above you can probably guess what our wish list is:

• Gearman should return memory to the OS when it is no longer needed. The argument here is that if you don’t want Gearman to use 2GB of memory, you can set a ulimit or take other measures to make sure you don’t ever get that far behind. That’s fine but in our case we would usually rather allow Gearman to use 2GB when it is needed, but we’d still like to have it back when it’s done!
• Workers should be better at balancing. Even if one server is far behind it should not be able to monopolize a worker to the detriment of all other job servers.
• Workers should be more aware of timeouts. Workers should have a way to remember when they failed to connect to a server and not try again for a configurable number of seconds. Or connections should be established to all job servers in a non-blocking manner, so that one timing out doesn’t affect the others.
• Servers should be capable of replication/aggregation. This is more of a want than a need, but sometimes it would be nice if one job server could be configured to pull jobs off of other job servers and pool them. That way jobs could be submitted over localhost on each FEN, but aggregated elsewhere so that one worker could process them in serial if rate limiting is desired, without potentially being slowed down by a malfunctioning FEN.
• Reduce latency for submitting asynchronous jobs locally. Submitting asynchronous jobs over localhost is fast, but it could probably be even faster. For example, I’d love to see how it could perform using unix sockets.

Even with these niggles and wants, Gearman has been a great, reliable and performant product that we are able to rely on to help keep the site fast and reliable for our users.

Supervisord

When talking about Gearman, I would be in remiss if I did not mention Supervisord, which we use to babysit all of our workers. Supervisord is a nice little python utility you can use to daemonize any process for you, and it will handle things like redirecting stdout to a log file, auto-restarting the process if it fails, starting as many instances of the process as you specify, and automatically backing off if your process fails to start a specified number of times in a row. It also has an RPC interface so you can manage it remotely, for instance if you notice a backlog of jobs piling up on one of your gearman servers, you can tell supervisord to fire up another 20 workers.

I hate the idea that this blog might be turning into nothing but a journal of all the things at Grooveshark that have ever broken, but some of the most interesting challenges we face are when things go terribly awry, so I’m not going to avoid talking about it just because it involves something breaking at Grooveshark, again.

What happened was, out of the blue IE8 could no longer run the site. Users were getting a message about making sure they did not have flash block enabled, which means the swf was failing in some way. We determined that the swf was in fact loading, so why was it lying to us? There is one file that swfs need in order to talk to other domains: crossdomain.xml. If that file fails to load, the swf isn’t going to work. I suspected that was happening in this case, so I loaded up http://cowbell.grooveshark.com/crossdomain.xml and IE complained that it wasn’t valid XML. View source showed me that IE was right. It was in fact what looked like binary garbage. Loading the same file in Firefox and Chrome worked perfectly fine, but IE8 on 4 different computers all showed the invalid XML.

Some months ago, we switched from serving pages up directly from Apache, to running Nginx in front of Apache as a reverse proxy with caching. The difference that made on our front end servers in terms of memory usage and CPU load is phenomenal. Although Nginx serves 30% of requests from cache now, the drop in server load was much more than 30%. Nginx is truly a wonderful addition to our http stack…but as you’ve guessed by now, it played a key role in the latest breakage at Grooveshark.

Force clearing the cache in IE8 and in nginx would sometimes fix the file, but not always. I then turned to wget and found the same thing: whenever the file was broken for IE8, it was identical in wget. wget was showing the exact same file size that Firebug was showing, which was the biggest clue: Firefox received the file gzipped because it supports deflate, but wget also received the file gzipped even though it doesn’t support deflate. My theory, which proved correct, was that IE8 was for some reason asking for the non-gzipped version, but receiving the gzipped version and barfing.

Why would that happen? Well, remember that we are using nginx as a reverse proxy cache. It turns out that we just recently added some auto-gzipping for certain file types to apache. What was happening was, nginx would get a request for a file not in its cache, and forward along the request (with all headers intact) to Apache. If this request came from a client that supports deflate, Apache would respond with a gzipped file. Nginx would store that gzipped file in its cache, and the next request that came in asking for that file, with or without deflate support, would get the gzipped version served up.

The fix was relatively simple: add a variable in nginx conf tracking whether or not the current client supports deflate. Append the value of that variable to the proxy key, meaning that gzipped and non-gzipped versions of the files will be cached separately, and served appropriately depending on what the client supports.

What’s not clear to me at this time is why IE8 would refuse to accept gzipped content for that file, and whether that applies to all .xml files in IE8…but at least it helped us catch what would have otherwise been an extremely obscure issue!

I’m posting this mostly as a note to myself.

When trying to build the pecl sphinx extension, ran into problems because I could not build libsphinxclient.

It looks like the bug causing that problem has been around for at least a year, but at least the fix is simple:
change line 280: void sock_close ( int sock ); to static void sock_close ( int sock );

Note: this applies to sphinx 0.9.9, the last “stable” release at the time of this writing.

Google has added support for crawling AJAX URLs. This is great news for us and any other site that makes heavy use of AJAX or is more of a web app than a collection of individual pages.

We have long worked around the issue of AJAX URLs not being crawl-able by having two versions of our URLs, with and without the hash. Users who are actually using the site will obviously get AJAX URLs like http://listen.grooveshark.com/#/user/jay/42, but if a crawler goes to http://listen.grooveshark.com/user/jay/42 they will get content for that page as well, while real users will be automatically redirected to the proper URL with the hash. Crawlers aren’t smart enough to go there on their own of course, but we provide a sitemap, and all links we present to crawlers are absent of the hash. Likewise, when users post links to Facebook via the app, we automatically give them the URL without the hash so they can put up a pretty preview of the link in the user’s news feed. The problem is, users also like to share by copying URLs from the URL bar. If users post those links anywhere, crawlers don’t know how to crawl them, so they either don’t or they just count it as a link to http://listen.grooveshark.com/ which isn’t great for us and is lousy for users too.

Google’s solution is for sites like ours to switch from using # to using #! and then opting in to having those URLs crawled. The crawler will take everything after the #! and convert the “pretty” URL into an ugly one. For example, /#!/user/jay/42 presumably becomes something like /?_escaped_fragment_=%2Fuser%2Fjay%2F42 when the crawler sends the request to us.

This is annoying and frustrating for several reasons:

1. All our URLs have to change
   We have to change all URLs to have a #! instead of just a #. This not only requires developer effort but makes our URLs slightly uglier.
2. All links that users have shared in the past will continue to be non-crawlable forever.
3. We now have to support 3 URL formats instead of 2

One of those URL formats no human will ever see; we are building a feature solely for the benefit of a robot.

Again, we greatly appreciate that Google is making an effort to crawl AJAX URLs, it’s a huge step forward. It’s just not as elegant as it could be. It seems like we could accomplish the same goals more simply by:

1. Requiring opt-in just like the current system
2. Using robots.txt to dictate which AJAX URLs should not be crawled
3. Allowing webmasters to specify what the # should be replaced with
   In our case it would be replaced by nothing, just stripped out. For less sophisticated URL schemes that just use #x=y, webmasters could specify that the # should be replaced by a ?

That solution would have the same benefits of the current one, with the additional benefits of allowing all crawling permissions to be specified in the same place (robots.txt), automatically making links already in the wild crawl-able, without requiring the addition of support for yet another URL format.