RSS
 

Simple PHP lockfile/pidfile implementation

30 Oct

We have a some PHP files that run as cron jobs. Some of them run frequently, but occasionally take longer to run than the interval we have them set to. To prevent pile-up, I created a simple lockfile mechanism. I take advantage of posix kill 0 to check if the script is still running, so I think this is only guaranteed to work on linux, definitely does not work on Windows last I checked.

All you need are two functions:

function lockfile_open()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        $pid = file_get_contents($pidfile);
        $running = posix_kill($pid, 0);
        if ($running) {
            return false;
        } else {
            unlink($pidfile);
        }
    }
    $handle = fopen($pidfile, 'x');
    if (!$handle) {
        return false;
    }
    $pid = getmypid();
    fwrite($handle, $pid);
    fclose($handle);
    register_shutdown_function('lockfile_close');
    return true;
}

function lockfile_close()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        unlink($pidfile);
    }
}

To use it, simply call lockfile_open(); at the beginning of cron execution and check the return, if it returns false then you have another copy of your script already running and you need to handle appropriately (in our case we log an error and die).

As with my previous post, I’ll break this down line by line to explain what is happening.

$pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';

this sets the path for the lockfile/pidfile. I am storing them in /tmp and basing the name entirely on the name of the script that is executing. Depending on your system you may need to store them somewhere else and come up with a different naming scheme.

if (file_exists($pidfile)) {
    $pid = file_get_contents($pidfile);


we check to see if our pidfile already exists. If no other copy of the script is running, it should not exist because the shutdown handler deletes the file. If it does exist then we need to read the contents of the file to get the pid, so we can check to see if it’s really still running, or if it’s stale.

$running = posix_kill($pid, 0);

doing a posix kill with signal 0 does not actually send a signal to the process, but still runs error checking, which includes testing to see if the pid is running.

    if ($running) {
        return false;
    } else {
        unlink($pidfile);
    }
}

if posix_kill returned something truthy then the process is in fact still running, so we return false to indicate that. If the pidfile is actually stale, we delete it and then continue as if the file never existed.

$handle = fopen($pidfile, 'x');

We create the pidfile with mode x, this will fail if the pidfile already exists, which means this handles the possible race condition of two scripts executing at the same time, where both check above for the pidfile and both see it as not existing. mode x guarantees that one of those scripts will succeed in creating the file and the other will fail.

if (!$handle) {
    return false;
}

If we failed to create the pidfile then we assume it’s due to the aforementioned race condition, so return false. NOTE: this assumes permissions are set up correctly for wherever you are saving your pidfile, if you set up a path that your crons can’t write to, this will fail to create the pidfile and assume it was due to the race condition even though there is probably not another instance of your script running.

$pid = getmypid();
fwrite($handle, $pid);
fclose($handle);

Here we get our current pid and write to to the pidfile, then close the file.

    register_shutdown_function('lockfile_close');
    return true;
}

If we made it this far then we are the first/only instance of the script currently running. We register our lockfile_close() function to clean up our pidfile when the script stops executing, and we return true to indicate that it’s safe to proceed.

function lockfile_close()
{
    $pidfile = '/tmp/' . basename($_SERVER['PHP_SELF']) . '.pid';
    if (file_exists($pidfile)) {
        unlink($pidfile);
    }
}

lockfile_close gets called automatically by the shutdown handler when the script stops executing. We check to make sure the pidfile still exists and then delete it, so we aren’t leaving stale pidfiles around.

 
 

Tail error logs to slack for fun and profit

12 May

We love slack, so we thought, wouldn’t it be great if our error logs posted to an #errors channel in Slack? This is obviously a very bad idea if you have noisy error logs, but for us we try to keep the chatter in the logs down so that people will actually pay attention to the errors, and it’s working out nicely, but it was a little complicated to get it set up so I thought I’d share.

First we installed and set up https://github.com/paulhammond/slackcat on our servers using the instructions there. It’s pretty simple and straightforward to get going, which is awesome!

Then we created two files on each server we wanted to monitor:
watchlogdir.sh

#!/bin/sh
trap "pkill -TERM -g $$; exit" INT TERM EXIT
while true; do
/path/to/tailtoslack.sh &
PID=$!
inotifywait -e create /var/log/xxx/
pkill -TERM -P $PID
kill $PID
done

tailtoslack.sh

while true; do
FILETOWATCH=$(ls -t /var/log/xxx/*.log | head -1)
tail $FILETOWATCH -f -n1 | grep -v "DEBUG:|^$" --color=never --line-buffered | /root/slackcat
sleep 31
done

So let’s look at what this is actually doing line by line:
trap "pkill -TERM -g $$; exit" INT TERM EXIT
This says, if we get any of the signals, INT TERM or EXIT, execute “pkill -TERM -g $$; exit”. Ok, well what does THAT mean? That utilizes the pkill command to send the TERM signal to all processes in the process group $$, and $$ means “my PID”. So essentially, we send TERM to all the processes that we spawned, so if we get killed, we don’t leave zombie children lying around.

while true; do
pretty self explanatory. all the code inside this block, we will try to do forever until we are killed

tailtoslack.sh &
this spawns tailtoslack.sh and runs it as a background process. tailtoslack needs to basically always run forever and we don’t want it to block what we’re about to do next

PID=$!
this grabs the pid of tailtoslack.sh and stores it in a variable called PID

inotifywait -e create /var/log/xxx/
this utilizes inotifywait to watch the /var/log/xxx/ directory for new files added. Our log files are written in the format of /var/log/xxx/yyyymmdd.log so when the day changes, a new file is written and that means we want to start tailing that log file and ignore the old one. inotifywait is blocking, so our code will sit here waiting for the new log file to be written, doing nothing until that moment.

pkill -TERM -P $PID
if we got here that means we have a new file, so we want to kill the old tailtoslack.sh and start the process over again. so this line is sending the TERM signal to all the processes owned by the old tailtoslack.sh

kill $PID
now we send the kill signal to tailtoslack.sh as well.

done
That’s it! on to the next file:

while true; do
this time we need a while loop because sometimes slackcat will exit (bad response from slack, or if you send it whitespace), but we don’t actually want to give up so we loop

FILETOWATCH=`ls -t /var/log/xxx/*.log | head -1`
this finds the most recently modified log file, ls -t sorts by most recently modified and “head -1” grabs the first line of output, then we store that in a variable called FILETOWATCH

tail $FILETOWATCH -f -n1 | grep -v "DEBUG:\|^$" --color=never --line-buffered | /root/slackcat
here we tail the file that we just determined was the most recently modified one, then we strip out lines starting with DEBUG: (since we don’t care about them) as well as empty lines (since empty lines crash slackcat), then we have to tell grep to not colorize the output and to buffer the output so it only sends complete lines to slackcat, since slackcat sends a message per line.

sleep 31
if we get to this line it means something in the tail pipleine on the previous line crashed. we don’t know why, but we’re hoping whatever condition caused the crash will pass soon, so we take a nap before we iterate through the loop again

done
we crashed, we took a nap, time to start over

that’s it! I am by no means a bash expert so it’s possible that some of this could be done better, but it works, and has been surprisingly robust!

For bonus points, here’s how I added it to systemd in CentOS 7:

Create file: /etc/systemd/system/logwatch.service :
[Unit]
Description=Watch web log files and pipe to slack
After=php-fpm.service

[Service]
ExecStart=/usr/bin/bash -c '/path/to/watchlogdir.sh'
Type=simple
User=youruser
Group=yourgroup
Restart=always
LimitNOFile=4096

[Install]
WantedBy=multi-user.target

Then you just need to run:
systemctl start logwatch

and once everything looks good:
systemctl enable logwatch

 
No Comments

Posted in Coding

 

South Carolina’s flag is bad for business

20 Jun

South Carolina’s confederate flag has come under intense scrutiny due to the recent actions of a certain racist asshole. Apparently last year, Governor Nikki Haley defended the confederate flag saying that CEOs do not have a problem with it.

I have two observations to make about this:

1. If a CEO is talking to the governor about moving their business to SC or has already moved their business to SC, they most likely had already decided the confederate flag was a non-issue for them, so why would they bring it up to the governor? The fact that CEOs aren’t talking to her about it doesn’t seem to say a whole lot about how CEOs in general feel about it.

2. Even if CEOs have no problems with the confederate flag, I know that SC is losing talent because of it. I know this because I know 4 extremely talented designers and software engineers from SC who left because of the blatant state sanctioned racism including the confederate flag. I also know that the confederate flag keeps talent away because although SC has the ideal climate for me and meets my requirements of being coastal, I refused to consider any jobs in that area because of the confederate flag. I will not tolerate living anywhere that is OK with prominently displaying a blatant symbol of racism. So even if CEOs are not complaining about the flag, it’s bad for business because it’s driving talent away, and businesses need talented people.

Now, I know some people claim that the confederate flag is not a symbol of racism and it’s about heritage, or it has some other meaning I’m not familiar with (I was not raised in the south, so I admit I may be ignorant on these things). Let’s assume that they are right and for some people the confederate flag has nothing at all to do with racism. I would argue that it does not matter what the flag means to some, if to the majority of people it represents racist ideology. Take the swastika: it had different meanings to different cultures, but you don’t see non-racists flying swastika flags and defending it as the ancient buddhist symbol of eternity, do you? No. Since the confederate flag is so closely associated with racism, it’s time for the non-racists who use it to mean something about heritage (or whatever they think it means) to pick a new symbol that does not have the hateful overtones of the confederate flag, unless they really don’t mind being associated with white supremacy…in which case I would argue that they are more racist than they think they are.

 
 

Android Wear: great now, awesome soon!

15 Jul

As everyone who pays attention to tech probably knows by now, Google gave away the LG G and Samsung Gear Live watches at I/O, as well as early access to the Android Wear companion app. I went with the LG G watch and have been using it since then.

The Watch

This post is mainly going to be about Android Wear, but I will share a little about the LG G watch first since I’m sure some people are curious. I actually think the Samsung is a much nicer looking watch with the curved edges vs the LG’s blocky square look. At first glance the LG actually made me nervous that it was going to stick out too far and I would end up catching and hitting it on things, but early reports of poor battery life on the Samsung convinced me to go with the LG. I also figured that if I changed my mind the higher retail price of the LG would make it easier to sell and replace with the Samsung later.

I’m happy to report that while it’s not the most stylish watch ever, it’s not so chunky that it gets snagged on things, so far it has never been in the way.

The battery life does seem to be significantly better compared to the what my friends with the Samsung are seeing, especially on the first night when they were new and we couldn’t stop playing with them, their watches died when mine still had over 60% battery life remaining, despite their having immediately charged theirs when they got them (I didn’t have time to top mine off). Under normal circumstances they have had no problem with the watch lasting a whole day, but my this is what the battery is like when it’s new; as it ages it may indeed have difficulty staying alive for a whole day, especially if I’m right and they end up getting a lot more use than we initially anticipated (more on that later). Obviously comparing battery life is difficult because everyone’s usage patterns are going to be different, but I should mention that we all had at least turned off the always-on feature, requiring a flick of the wrist to turn on the display, so it was as close to an apples-to-apples comparison as we could get under the circumstances. I’ve since switched my display to always-on, and still have plenty of battery left over at the end of the day.

Even though I am generally happy with my decision to go with the LG G, I do have some complaints, some of which may be eventually fixed by software:

Notifications – There is no way to change the length or intensity of the notification buzz. The default notification buzz is so short and gentle that I often do not feel it. The buzz for an incoming phone call or alarm is stronger and more persistent so I have no trouble feeling those, so I would love to be able to increase the strength of the buzz for a normal notification. I do not know if Samsung allows you to customize this, or what the default buzz is like.

Screen on gesture – Turning on the screen with a flick of the wrist is not as elegant as it sounds. It does not seem to be controlled by a gesture or motion at all, — This complaint has actually already been fixed by an update since I started writing this blog post! It is indeed now controlled by an actual gesture, but it’s still not perfect: about half the time when I flick my wrist the gesture isn’t recognized, especially if the strap is a little loose. There’s also still a magical angle that, if you hold the watch at exactly that angle the screen is on, no matter what. You can turn the screen off and it will just turn right back on even if you are perfectly still. It’s a minor annoyance though, the worst case if the gesture is not recognized is that you try again, or touch the screen to turn it on.

The band – The wrist band material is uncomfortable. I may be in the minority but I really strongly dislike the synthetic rubber material that both the LG and Samsung watch bands are made from. It traps sweat, is generally uncomfortable, and eventually gives me a rash. Not to mention that it looks cheap to me. Fortunately, the LG accepts a standard band so I replaced it with this one which I’m happy enough with (but any 22mm band will work). I’m told the Samsung can accept a standard band as well, but I think it would look really bizarre because of the way the band integrates with the watch.

Charger – The charger is completely nonstandard. Not a big deal in general for the LG G since the battery life is generally pretty good (with the screen set to auto-off, I’m usually around 80% battery life before I go to bed, with always on about 50%), but it will be annoying to have to take another proprietary charger with me when travelling. The charger at least accepts a standard micro USB connector, so it’s really just another random dongle to lose in my bag. At home the cradle is certainly more convenient than micro USB would be, but it would be nice to be able to charge with the cradle at home and fall back to standard micro USB on the road, especially since the cradle includes an adhesive to hold it in place on your nightstand. The Samsung also has a proprietary cradle charger thing, and I’ve seen reports of the clip-on tab breaking easily, which can’t happen with the LG since it’s only held on by magnets.

Charging at night – The screen is always on when it’s connected to the charger. I was strongly considering keeping the charger in another room because the amount of light from the screen is enough to be annoying, but I eventually realized that the watch charges so quickly that there’s really no need to charge it overnight. Now I actually turn the watch off when I go to sleep, and pop it on the charger in the morning, which automatically turns it back on. By the time I’m done with my morning routine, the watch is fully charged.

Android Wear

On to what this blog post is actually about: Android Wear. I have to admit that if I didn’t get an Android watch for free, I would have passed up the first generation and waited to see if there was something truly exciting that I could do with them. Despite being psyched about the wearable computing movement, I just didn’t find the smart watch a very compelling application, with such a small screen and limited forms of input, it just seemed pointless. But even after my few days with the LG G, I was sure that if something happened to it, I would buy a new one in a heartbeat.

Why it’s great right now

Some of my favorite features:
Timer/reminders – I can say “ok google set a time for 30 minutes” or “ok google remind me to write a blog post at 4:30pm” and I get a buzzing reminder on my watch at the appropriate time. It’s not only super convenient but it’s way less disturbing to other people than a noisy alarm on my phone, and unlike a buzzing reminder on my phone, it’s completely impossible to miss when this is going off. I would like if it was possible to manage these timers/reminders from my phone, because part of the appeal of not disturbing people sometimes is also not having to loudly talk to my watch.

Agenda – Every night a card pops up remind me of tomorrow’s agenda from my calendar. This is great because I’ve actually never been good about consistently checking the night before to make sure I am prepared for the next day. Now it’s automatic. This may not work for people who go to bed before midnight because right now it seems to just be based on the date; the card pops up at exactly midnight. Since I’m a night owl, that’s more than enough time for me. If you want to see your agenda any time when the card doesn’t automatically pop up you just have to say “show agenda” to see it.

Navigation – Currently the only useful use case for this is when you are walking around in a strange city, but when you are it’s so nice. I feel much less conspicuous glancing at my watch periodically than walking around with my phone out. I feel this app could use some visual tweaks as it’s very plain/ugly right now, and I’d love for them to find a way to fit a little more information on it (a way to see the next step, landmarks, alternative street names) but even as-is it’s highly useful for walking.

Weather – Weather was the one thing I consistently went to Google Now on my phone for. Now it’s on my watch, usually right under the time. A swipe to the right shows the weather for the next 4 days. I’d love to see them add support for another swipe to show me the hourly breakdown for the next 24 hours like I can in Google Now on my phone, that information is often more useful to me than what the weather is going to be like in a couple of days.

Messaging & hangouts – It’s nice being able to read most SMS messages at a glance, but what’s really cool is how much less annoying it is to be in a group hangout with a bunch of chatty people while you’re busy doing other things. No more constantly having to pull out your phone to keep up with the conversation. It’s also really handy to be able to send a text message when my hands are otherwise occupied/dirty, like when I was in the middle of cutting up raw chicken and needed my girlfriend’s help with something but she was outside tending the garden, or like when I was kayaking and wanted to let her know I was getting close to where we were meeting up. Did I mention that the watches are water resistant?

Grooveshark & other media players – Grooveshark works right out of the box on this, even though our most recent release predates the existence of Android Wear. Once music is playing, a card appears on the watch with the album art along with artist and song names, with a Play/Pause button. A swipe to the right shows a Next button, and another swipe to the right brings up a Previous button. All of these work, right now with no work necessary on Grooveshark’s side. The only annoyance I have with this is that the card doesn’t come up until music is actually playing. So I can’t say “ok google, start Grooveshark” and then immediately control playback. I still have to get it started playing on my phone. But after that, controlling via the watch works perfectly. There are two use cases this is awesome for: When I’m cooking in the kitchen, I have my phone plugged into a small speaker system playing music. I can now see what is playing, and control playback, without having to walk over to it and unlock my phone. The other use case any time when I am listening to music on my bluetooth headset and my phone is in my pocket – I no longer have to fish it out of my pocket to control playback…that makes the whole experience so much more convenient in general, but it was especially nice on the insanely long flight back to Gainesville from San Francisco; fishing anything out of my pockets when sitting in a cramped airplane seat is a chore.

Netflix & Chromecast – Much like how you can control Grooveshark with the watch, you can also control Netflix when casting to Chromecast. I can just run into the kitchen whenever I want, and if it turns out I’m missing something I can instantly pause it. Previously if I wanted to have that option I would have to make sure to bring my phone with me, then turn it on, unlock the screen and finally hit pause…by which point I’ve already missed whatever I was trying not to miss, and of course half the time I’d forget to bring my phone with me anyway. It’s one of those things that I never really thought of as annoying before but I find myself greatly appreciating the watch for solving anyway. I do wish they had the “skip back 30s” option like the app does on the phone, and there is a minor bug that I expect will be fixed soon where the card stays on the watch after the show has finished playing and it’s impossible to dismiss until you toggle play/pause on the watch, even though it’s not even playing (even if the TV is off!)

How it can be even better
Even though Android Wear is great today, I’m especially excited for the future because it has so much potential to be thoroughly awesome. Here’s hoping even a fraction of these come true.

Better Casting: The way Netflix & Chromecast works, anyone in the house can control playback no matter who started it. Right now though, if my girlfriend starts something on Netflix, I have to start up the app on my phone, switch to our shared profile, turn on casting, wait a few seconds for it to figure out what is going on, and then I can finally pause. If Android Wear could just figure out that hey, something is playing on the Chromecast, so you should have control options, that would be truly awesome. Obviously the same would be true on the Android phone, but my expectations for awesome context awareness as so much higher for Android Wear because it already delivers that experience for some things.

Payments – I want to stop carrying a wallet, or at least credit cards, and I don’t think NFC is ever going to get us there: retailers need to buy new equipment, more than just flagship phones need to support it, and a standard that everyone can use and be OK with needs to emerge. Imagine if whenever you walked into a store you got a card on your watch along the lines of: “It looks like you’re in Starbucks. Swipe right to pay.” swiping right shows a one-time-use QR code that the retailer can scan. That’s a system that I think would actually have a chance of succeeding because it utilizes equipment that retailers already have, and doesn’t require users to fiddle around with anything, the information is just there when they need it. If, instead of Google Wallet, that was supported directly by Visa, MasterCard, etc., the friction to adoption would just melt away. In fact Visa and MasterCard might have a strong incentive to go along with it since the one time use nature could reduce fraud by eliminating stolen credit card numbers. The same technology could and should exist on the phone as well, but it would be truly convenient on the watch.

Where’s my phone?! – Maybe it’s just me but when I’m at home I have a penchant for leaving my phone in odd places and then forgetting where that was. If the watch is in range (and the range is quite good for bluetooth), I’d love to be able to make my phone buzz or ring from the watch.

Better media player support – As far as I can tell there’s no way to send a Play event like you can from bluetooth media devices. For example in most bluetooth enabled cars there’s an option for it to just send play to your phone the moment you turn the car on. I don’t want that to be automatic with my watch, but I’d love to be able to say “ok google, play” and have it send a play event to whatever media player I had open most recently (usually Grooveshark of course!)

Soundhound/Shazam – I’d love to be able to say “ok google, what’s playing right now?” and get an answer on my watch.

Fitness / health / sleep tracking: Some of this will have to come on a different watch from either of the ones available today, because even the Samsung doesn’t have continuous heart rate monitoring. There’s no denying that fitness is a killer app for wearables, and one that’s more or less ignored in this version. It does include step counting, but that’s it. Continuous heart rate monitoring, sleep tracking, waking you up at just the right time, and food/calorie monitoring are all good places to start, and it’ll be interesting to see what Apple’s take on this will be since rumor has it their watch will be health focused.

Speaker This one is tricky because one of the things that nice about the watch most of the time is that it is silent. But there are times where it would be really nice if it could read something back to me, or even reading it back to me via the phone’s speaker would be an improvement.

Wireless charging – The charger for the LG isn’t bad or too annoying, but the magnet isn’t really strong enough to guarantee that it seats properly without fiddling. What would be really nice is being able to just take my watch off and plop it on a tray without paying any attention to where or how it lands.

For the love of god let me turn off the screen when charging! – My current solution of turning the watch off at night feels weirdly old fashioned, and it’s not exactly easy, you have to dig through the settings to find the option to turn it off. Really I just want to be able to put the watch on its cradle at night and turn the screen off, there’s no reason for it to be on lighting up the whole room when I’m trying to sleep.

Make everything dismissable – Certain notifications are impossible to dismiss (like that Netflix one I mentioned earlier). It really should be possible to tell every single notification to fuck off if I really want to. Go ahead and make it a 3 step process, as long as it’s possible.

Pin/resurrect cards – At the same time I also occasionally find myself accidentally dismissing cards when I actually wanted them to stick around and was just trying to get back to the “main” screen. A stronger hint that you’re about to dismiss a card would help, but really it would be nice if there was a way to quickly undo. It’s especially bad because dismissing a “Google Now” card seems to teach it that you don’t want that card anymore. I accidentally dismissed the weather card and I had to look up the weather a few times a day for a couple of days before it started coming in on its own again.

Better watch faces – I know it’s super early, but for now I’m stuck with a black watch face because I wanted one that shows the time and date, and none of the colorful ones do that. I’m sure it’s only a matter of time, there have already been a bunch of 3rd party watch faces made so I’m sure it’s only a matter of time before the perfect one for me exists.

Camera I’m officially in fantasy land on this one but some day in the distant future it would be cool to be able to take pictures straight from my watch. This was inspired from the most recent of many attempts to take a picture of an interesting creature in my back yard — by the time I got my phone out and fired up the camera app, it was gone. The biggest impediment to ever getting this is the creepy/spywatch factor, and the last thing Google needs is the same set of concerns people have about Glass.

Multiple device controls – This isn’t entirely Android Wear specific, but really in all of Android if you have multiple devices tied to the same account, dismissing a notification on one should dismiss it on all the others. My tablet constantly ends up with tons of useless notifications on it that I have already seen on my phone. Android wear has raised my awareness to the possibility of dismissing a notification on one device affecting the other, so now we just need to make it do that from my watch to all my devices, rather than just the phone it’s paired to.

Desktop integration – It would be really awesome if my watch could be more than just a remote extension of my phone. If there was a chrome extension, Android Wear would become even more useful: Controlling YouTube, Grooveshark, Netflix when they are running in the browser too? Yes please! I’d also love to see the same automatic lock/unlock functionality that’s coming to Android L, on my windows desktop. Even if it meant pulling up some card on my watch to lock the screen (or check whether it’s locked), that would be great.

3rd party integrations – Hangouts has me spoiled. Whenever I get a Facebook notification about a comment for example, I expect to be able to actually read the comment by tapping on the notification or swiping to the right, but right now all I can do is choose “Open on Phone” and then go find my phone. When app developers start including the full context in their notifications, they will be incredibly useful.

Wrapping it up
Android Wear is a great platform today, with a lot of potential to become truly amazing in the very near future. The watches available today are not perfect, but their drawbacks are minor, at least in the case of the LG G which I have experience with. If you have any interest in wearable computing or just making your life a bit more convenient, I think picking one up today is well worth it.

 
 

Google Analytics: Drop in Safari Visits

28 Sep

Starting around Thursday, 9/19/2013, Google Analytics has been showing about 50% fewer visits from Safari users than the Thursday before that for Grooveshark. (Aside: by default we compare each day to the same day of the week from the previous week because usage patterns for when people are on Grooveshark are highly dependent on what day of the week it is. The way people use Grooveshark is most wildly different comparing weekdays to weekends, but even a Monday session is very different from a Thursday session)

I suspect it’s related to a longstanding issue as explained in these 2 blog posts:

http://sunpig.com/martin/archives/2010/01/08/how-to-detect-a-page-request-from-safari-4s-top-sites-feature.html
(2010!)
http://blog.splitwise.com/2013/05/13/safari-top-sites-pollutes-google-analytics-and-mixpanel/ (2013)

it appears that the GA team is finally detecting these “preview” loads and not counting them, although I have not been able to get official confirmation of that.
The evidence:

It’s not just happening to Grooveshark:
Google Analytics Forum question
Superuser question (deleted: google archive)
Another forum post asking about the drop

  • Average visit duration is up by 163% for Safari users. That makes sense because the Safari preview visits would by definition be extremely short.
    Our bounce rate for Safari users is down by 82%. We had previously determined that Safari preview visits were all being counted as bounces, artificially inflating that number for Safari users.
  • I also dug into different Safari versions, and at least for the 3 most popular versions on Grooveshark (all that I checked), the % drop in visits is the same, which strongly implies that this drop is not due to some recent update from Apple. We also tested the site to make sure that everything still appears to work in Safari, and we have not seen an uptick in complaints from Safari users which we would expect to see if we had introduced a Safari-breaking bug.
  • New vs Returning: GA reports that “new” Safari visitors are down 36%, while “returning” Safari visitors are down a whopping 57%. This also points towards the Safari preview issue because users who visit Grooveshark more frequently are less likely to be considered new and are more likely to have Grooveshark show up in Safari preview.
  • Pages/Visit by Safari users is up a whopping 76%. This makes sense since a “preview” visit would only hit the homepage, but real users are very likely to visit other pages.

Note: Grooveshark’s numbers are generally considered proprietary, which is why all the stats above are in relative percentages. Sorry if that made it harder to follow.

I have reached out to Google Analytics via twitter and also commented on the Google Analytics discussion thread, but we have not heard any official word yet. Ideally this would be the sort of thing we’d hear about from Google directly before it happened. Barring that, it would be nice to see a blog post from them about it, which also hasn’t happened yet.

Unrelated:
On the off chance that I now have a Googler’s attention: a probably unrelated issue that we see for the day of Friday, 9/20/2013 is for that day our stats are mysteriously low and internally inconsistent. Visits for that day are 50% lower than the previous Friday. Our internal bandwidth and user activity #s look normal for that day. If I look at New vs Returning, we had significantly more returning visitors than we had total visits…how could that even be possible?! We have seen anomalies like that before but they usually only appear for a short time and are corrected a day or so later.

 
 

Git Tips (just a couple)

05 Jun

Below are a couple of git tips for the advanced git user. This is not meant to be remotely comprehensive in any way, just a couple of things I have figured out that I haven’t seen floating around out there yet.

Always rebase

At Grooveshark, we like to keep our history clean, such that the history of the main repo reflects when code was actually committed/accepted, regardless of when that code was written. As such, we almost always want to rebase. If you search for how to rebase by default, most of the advice you’ll find says to add the following to your ~/.gitconfig:

[branch]
    autosetuprebase = always

What this does, however, is it tells git to set up each branch to rebase as you create it. Any branch that git already knows about will not be set up to rebase; you’ll have to do that manually. Then in the future if you change your mind and decide you don’t want this functionality, you will have to remove it from each branch one by one. Instead put this in your ~/.gitconfig:

[pull]
    rebase = true

Now you’ve made the default behavior for pulling be to rebase. If you want to override that behavior for a particular branch, just set rebase = false in the .git/config file for the applicable repo, under the particular branch you want to do that for.

Github pull requests by the number

As mentioned earlier, when we do a pull request, we want it to be rebased onto the latest before it gets merged, so it’s a simple fast forward. Add the following to your ~/.gitconfig:

[alias]
    review = !sh -c 'REF=`git rev-parse --symbolic-full-name --abbrev-ref HEAD` && git fetch $1 $2:mergeReview -f && git checkout mergeReview && git rebase $REF && git checkout $REF && git diff HEAD..mergeReview' -
    pr = !git review origin refs/pull/$1/head
    approve = merge mergeReview

What’s going on here?
First we make a new command for reviewing arbitrary refs from arbitrary forks, called ‘review’. To use it, first checkout the target branch that you want to merge into, then run: ‘git review {remote} {ref}‘. Behind the scenes that fetches the remote ref into a temporary local branch called mergeReview, based on the remote ref you are reviewing. It rebases that onto the target branch you were in, switches you back to the target branch and then does a diff for you to review.
Next we make a new command for reviewing pull requests from github, called ‘pr’. This really just saves you from having to remember where github keeps pull request refs. It just calls git review and passes in the remote ref for the pull request.
Finally, ‘approve’ just merges the temporary mergeReview branch into the branch you’re currently in, which if you followed the instructions above is the target branch.

 

XSS cheat sheet

26 Apr

Discovered this amazing XSS cheat sheet while trying to prove to a co-worker that using a regex to prevent <script> tags embedded in HTML was not going to be effective. I knew about the UTF8 vulnerabilities and all of the obvious ones, but the US-ASCII Encoding one especially was new to me, impressive!

 
 

Migrating to github

26 Jan

I needed to migrate a couple of repositories with lots of tags and branches to github, and Github’s instrucitons just weren’t cutting it:

Existing Git Repo?
cd existing_git_repo
git remote add origin git@github.com:[orgname]/[branchname].git
git push -u origin master

That’s great as long as all I care about is moving my master branch, but what I really wanted was just to make github have everything that my current origin did, whether or not I was tracking it. A quick cursory Google search didn’t find any instructions how to do this, so I had to figure it out the old fashioned way, by reading git help pages. The quickest and easiest way I could find to do this after creating the repo on github is:

git clone git@[originurl]:[orgname]/[reponame].git [local temp dir] --mirror
cd [local temp dir]
git remote add github git@github.com:[orgname]/[reponame].git
git push -f --mirror github
cd ..
rm -rf [local temp dir]

that’s it! At this point github should have everything that your original origin had, including everything in /refs. It won’t have any of your local branches, since you did everything from a clean checkout. You might also want to change where origin is pointing:

cd [dir where your repo is checked out]
git remote add oldOrigin git@[oldgitserver]:[orgname]/[reponame].git
git remote set-url origin git@github.com:[orgname]/[reponame].git

Of course, this same procedure should work for moving from any git server to any other git server too.

 

Grooveshark is Hiring (Part 3 – Devsigner Edition)

26 May

Grooveshark is looking for talented web developers. For part 3, I’m listing our front end developer/designer (affectionately known here as “devsigner”) position.

Grooveshark Web Developer / Designers

Must be willing to relocate to Gainesville, FL and legally work in the US. Relocation assistance is available.
Responsibilities include:
Maintaining existing HTML templates and JavaScript code
Writing clean well-formed HTML and CSS
Writing JavaScript to be able to achieve the desired user experience
Using Photoshop and Illustrator and be able to convert images to markup and JavaScript
Optimizing HTML and CSS for speed considerations

Desired Qualities:
Enjoy writing high quality, easy to read, well-documented code
High level of detail given to converting designs
Able to follow coding standards
Good written and verbal communication skills
Well versed in best practices & security concerns for web development
Ability to work independently and on teams, with little guidance
Capable of prototyping features (either visually or concretely)
Capable of having a flexible schedule

Experience:
Experience with JavaScript, HTML & CSS
Experience in Photoshop and designing websites
Familiar with jQuery, including JavaScriptMVC
Familiar with Actionscript
Experience with version control software

Not Desired:
Uses WYSIWYG editors
Only searches for jQuery plugin that fits the job

 
 

Grooveshark is Hiring (Part 2 – Javascript Edition)

19 May

Grooveshark is looking for talented web developers. We are looking to fill a wide variety of web dev positions. The next few posts I make will be job descriptions for each of the major positions we are looking to fill. For part 2, I’m listing our Javascript developer position. If your skillset is more front-end leaning but you feel like you would be a good fit for Grooveshark, by all means apply now rather than waiting for me to post the job description. :)

Grooveshark is looking for a hardcore JavaScript developer

NOTE: Grooveshark is a web application, written with HTML/CSS, JavaScript, PHP, and ActionScript 3, not a web page or collection of webpages. This position involves working directly with the client-side application code in JavaScript: the “backend of the frontend” if you will.

Must be willing to relocate to Gainesville, FL and legally work in the US. Relocation assistance is available.
Responsibilities:
Maintaining existing client-side code, creating new features and improving existing ones
Writing good, clean, fast, secure code on tight deadlines
Ensuring that client architecture is performant without sacrificing maintainability and flexible enough for rapid feature changes
Striking a balance between optimizing client performance versus minimizing load on the backend
Integration of third-party APIs

Desired Qualities:
Enjoy writing high quality, easy to read, self-documenting code
A passion for learning about new technologies and pushing yourself
A deep understanding of writing bug-free code in an event-driven, asynchronous environment
Attention to detail
A high LOC/bug ratio
Able to follow coding standards
Good written and verbal communication skills
Well versed in best practices & security concerns for web development
Ability to work independently and on teams, with little guidance and with occasional micromanagement
More pragmatic than idealistic

Experience:
Extensive JavaScript experience, preferably in the browser environment
Experience with jQuery
Experience with jQueryMX or another MVC framework
HTML & CSS experience, though writing it will not be a primary responsibility
Some PHP experience, though you won’t be required to write it
Knowledge of cross-browser compatibility ‘gotchas’
Experience with EJS, smarty, or other templating systems
Experience with version control software (especially git or another dvcs)

Bonus points for:
Having written a client application (in any language) that relies on a remote server for data storage and retrieval
Having written a non-trivial jQuery plugin
Experience with JavaScript/Flash communication via ExternalInterface
Experience with integrating popular web APIs (OAuth, Facebook, Twitter, Google, etc) into client applications
Experience with ActionScript 3 outside the Flash Professional environment (ie, non-timelined code, compiling with mxmlc or similar)
Experience developing on the LAMP stack (able to set up a LAMP install with multiple vhosts on your own)
Experience with profiling/debugging tools
Being well read in Software Engineering practices
Useful contributions to the open source community
Fluency in lots of different programming languages
BS or higher in Computer Science or related field
Being more of an ‘evening’ person than a ‘morning’ person
A passion for music and a desire to revolutionize the industry

Who we don’t want:
Architecture astronauts
Trolls
Complete n00bs (apply for internship or enroll in Grooveshark University instead!)
People who want to work a 9-5 job
People who would rather pretend to know everything than actually learn
Religious adherents to The Right Way To Do Software Development
Anyone who would rather use XML over JSON for RPC

Send us:
Resume
Code samples you love
Code samples you hate
Links to projects you’ve worked on
Favorite reading materials on Software Engineering (e.g. books, blogs)
What you love about JavaScript
What you hate about JavaScript (and not just the differences in browser implementations)
Prototypal vs Classical inheritance – what are the differences and how do you feel about each?
If you could change one thing about the way Grooveshark works, what would it be and how would you implement it?

 

If you want a job:  jay at groovesharkdotcom
If you want an internship: internships@grooveshark.com