Comments on: Detect crawlers with PHP faster http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/ Rantings of a Grooveshark Developer Wed, 21 Sep 2011 10:35:10 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: www.syso.pl http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-829 www.syso.pl Wed, 17 Nov 2010 02:00:51 +0000 http://wanderr.com/jay/?p=166#comment-829 I'm using strpos instead of preg_match i thing it should be faster I’m using strpos instead of preg_match i thing it should be faster

]]> By: IWM - Marketing Internet http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-705 IWM - Marketing Internet Sat, 23 Oct 2010 15:25:20 +0000 http://wanderr.com/jay/?p=166#comment-705 Great post, it's going to help increase my crawler speed detection by a great deal. Is Bing's crawler integrated ? Great post, it’s going to help increase my crawler speed detection by a great deal.

Is Bing’s crawler integrated ?

]]> By: Mike http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-290 Mike Thu, 18 Jun 2009 23:42:32 +0000 http://wanderr.com/jay/?p=166#comment-290 Great post, very helpful. I added some more crawlers: Bloglines subscriber|Dumbot|Sosoimagespider|QihooBot|FAST-WebCrawler|Superdownloads Spiderman|LinkWalker|msnbot|ASPSeek|WebAlta Crawler|Lycos|FeedFetcher-Google|Yahoo|YoudaoBot|AdsBot-Google|Googlebot|Scooter|Gigabot|Charlotte|eStyle|AcioRobot|GeonaBot|msnbot-media|Baidu|CocoCrawler|Google|Charlotte t|Yahoo! Slurp China|Sogou web spider|YodaoBot|MSRBOT|AbachoBOT|Sogou head spider|AltaVista|IDBot|Sosospider|Yahoo! Slurp|Java VM|DotBot|LiteFinder|Yeti|Rambler|Scrubby|Baiduspider|accoona From http://www.httpuseragent.org/list/Robot, Spider, Crawler-c16.htm Also, maybe you should use the 'i' flag, e.g: $isCrawler = (preg_match("/$crawlers/i", $userAgent) > 0); To do case-insensitive matching? Great post, very helpful.

I added some more crawlers:

Bloglines subscriber|Dumbot|Sosoimagespider|QihooBot|FAST-WebCrawler|Superdownloads Spiderman|LinkWalker|msnbot|ASPSeek|WebAlta Crawler|Lycos|FeedFetcher-Google|Yahoo|YoudaoBot|AdsBot-Google|Googlebot|Scooter|Gigabot|Charlotte|eStyle|AcioRobot|GeonaBot|msnbot-media|Baidu|CocoCrawler|Google|Charlotte t|Yahoo! Slurp China|Sogou web spider|YodaoBot|MSRBOT|AbachoBOT|Sogou head spider|AltaVista|IDBot|Sosospider|Yahoo! Slurp|Java VM|DotBot|LiteFinder|Yeti|Rambler|Scrubby|Baiduspider|accoona

From http://www.httpuseragent.org/list/Robot, Spider, Crawler-c16.htm

Also, maybe you should use the ‘i’ flag, e.g:

$isCrawler = (preg_match(“/$crawlers/i”, $userAgent) > 0);

To do case-insensitive matching?

]]> By: Jay http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-275 Jay Mon, 27 Apr 2009 15:58:43 +0000 http://wanderr.com/jay/?p=166#comment-275 Honestly I'm not intimately familiar with the differences -- I tend to just use preg_match whenever I need to match a regex. I'll check it out! Honestly I’m not intimately familiar with the differences — I tend to just use preg_match whenever I need to match a regex. I’ll check it out!

]]> By: Cult-foo » Detect crawlers with PHP http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-274 Cult-foo » Detect crawlers with PHP Mon, 27 Apr 2009 09:59:25 +0000 http://wanderr.com/jay/?p=166#comment-274 [...] After reading this i decide to update my code a bit. Change is connected to usage of function on high volume [...] [...] After reading this i decide to update my code a bit. Change is connected to usage of function on high volume [...]

]]> By: elPas0 http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-273 elPas0 Mon, 27 Apr 2009 09:20:44 +0000 http://wanderr.com/jay/?p=166#comment-273 you are quite right about the loop but why use preg_match dont think about strpos() or strstr() instead ? it should be faster you are quite right about the loop
but why use preg_match
dont think about strpos() or strstr() instead ? it should be faster

]]> By: Jay http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-269 Jay Thu, 09 Apr 2009 01:44:54 +0000 http://wanderr.com/jay/?p=166#comment-269 @chris I'm not really sure how that would work, the set to match against is relatively small but the set of all user agent strings approaches infinity... @chris I’m not really sure how that would work, the set to match against is relatively small but the set of all user agent strings approaches infinity…

]]> By: Jay http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-268 Jay Thu, 09 Apr 2009 01:25:51 +0000 http://wanderr.com/jay/?p=166#comment-268 We've explored using memcache for sessions in the past but there are some issues with concurrency. If a user makes two requests back-to-back, each one might try to upcate the session. Memcache has no locking so one update will overwrite the other, leading to lost data. This can be mitigated somewhat by using sessions for less, or by breaking up the session data so that it's not one large serialized piece of data. /dev/shm isn't shared so that won't work for us for sessions either, but I would like to get us using /dev/shm for mysql temp space, so that filesorts can happen in-memory. We’ve explored using memcache for sessions in the past but there are some issues with concurrency. If a user makes two requests back-to-back, each one might try to upcate the session. Memcache has no locking so one update will overwrite the other, leading to lost data. This can be mitigated somewhat by using sessions for less, or by breaking up the session data so that it’s not one large serialized piece of data.

/dev/shm isn’t shared so that won’t work for us for sessions either, but I would like to get us using /dev/shm for mysql temp space, so that filesorts can happen in-memory.

]]> By: James Hartig http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-267 James Hartig Thu, 09 Apr 2009 01:13:14 +0000 http://wanderr.com/jay/?p=166#comment-267 You should try using memcache for your sessions and DB backups, in case the memcache gets erased (at shutdown). You could also just make a mount on /dev/shm and load the sessions into memory for faster access? -fastest963 You should try using memcache for your sessions and DB backups, in case the memcache gets erased (at shutdown). You could also just make a mount on /dev/shm and load the sessions into memory for faster access?

-fastest963

]]> By: Christopher Suter http://wanderr.com/jay/detect-crawlers-with-php-faster/2009/04/08/comment-page-1/#comment-266 Christopher Suter Wed, 08 Apr 2009 21:39:36 +0000 http://wanderr.com/jay/?p=166#comment-266 given the relatively static nature of the list, you could probably define a very small, very fast hashing algorithm with a small range set (ideally, the range would be the set {0,1,...num_crawlers}). Then it could be blazing fast! given the relatively static nature of the list, you could probably define a very small, very fast hashing algorithm with a small range set (ideally, the range would be the set {0,1,…num_crawlers}). Then it could be blazing fast!

]]>