Comments on: ORDER BY SLOW()

By: Matthew Montgomery

Matthew Montgomery — Wed, 28 Jan 2009 19:05:55 +0000

See: solutions at http://jan.kneschke.de/projects/mysql/order-by-rand/

By: Frederick R

Frederick R — Fri, 31 Oct 2008 11:30:26 +0000

It is different from ORDER BY RAND() in sense that you actually take only 10 rows given by mysql then order it by the randomize number ran.

Unlike ORDER BY RAND(), mysql will take all 6m tuples and randomize it.

One idea to improve the code would be to increase the samples and do another limit. Even better if you play with the result a little more…

SELECT * FROM (SELECT id,name,rand() as ran FROM table WHERE id like '345' OR name like 'exa' LIMIT 100) AS x ORDER BY x.ran LIMIT 10

Feeding SQL another random number or random string from your script will give an even more mixed-up result. Still a lot faster than RAND().

By: Jay

Jay — Fri, 31 Oct 2008 10:14:53 +0000

What that query will do is grab the first 10 rows in the table, and order them by rand. If you only want the first 10 rows to be considered, and your table is much larger, then yes this is faster. Otherwise you are just doing explicitly what MySQL does implicitly when you do an ORDER BY RAND().

By: Frederick R

Frederick R — Fri, 31 Oct 2008 06:41:45 +0000

Alternative random result; I think more flexible and faster result.

SELECT * FROM (SELECT id,name,rand() as ran FROM table LIMIT 10) AS x ORDER BY x.ran

Cheers!

By: Gabe da Silveira

Gabe da Silveira — Tue, 21 Oct 2008 01:27:14 +0000

A few notes:

First, if you are limiting the data set via a where clause, if the set is small enough, then ORDER BY RAND() may be fast enough for you.

Another thing to keep in mind is that:

SELECT id FROM table ORDER BY rand();

Can be orders of magnitude faster than:

SELECT * FROM table ORDER BY rand();

Due to the temporary table creation. Granted you probably will then need to do another query to pull the data, but…

Regarding multiple queries to the database, you should run benchmarks with your particular setup to find out if multiple queries will actually hurt you. In some cases what you’re trying to do can be more efficient in code than SQL even considering connection latency. On my setup even though the database is on a separate cluster, it is still very low latency and multiple queries have not been a problem.

With that in mind, I think the original fast solution you post is pretty good (and no less elegant to me than a weird subselect). When I started database programming 10 years ago I gravitated in the direction of as many efficient joins as possible to minimize queries. It worked pretty well, but after working on more complex applications I found the benefits of a nice ORM system and simple queries to ultimately be more beneficial for productivity without hurting performance (ie. knowing where to optimize).

Finally, another thing to watch out for in your solution is that in sparse tables the randomness can suffer severely. Imagine a table with rows 1-100. Delete 2-99 . Now you have a 99% chance of getting 100 and a 1% chance of getting 1.

By: Jay

Jay — Thu, 21 Aug 2008 06:42:23 +0000

Hi Andreas, unfortunately that’s where this solution falls apart. As soon as you’re dealing with a partial data set, there aren’t any elegant solutions to the problem (that I know of). If you do find something that works, is quick and elegant, please do share!

By: Andreas

Andreas — Wed, 20 Aug 2008 23:05:13 +0000

I’m a newbie but this looks like a good and fast solution.
My problem is that I want a random row but also only select a row with a specific country/age/gender etc..
Is it possible to include a WHERE statement somewhere in this? I’ve tried but can’t get it right.. thanks!

By: Jay

Jay — Tue, 05 Aug 2008 04:31:35 +0000

Hi Patrick, sure I understand, we’ve all been there before. :)

I assume you mean this part: SELECT * FROM Table T – T is an alias for ‘Table’ in this case. I was just being lazy, the longer, more obvious way to type the same thing would be SELECT * FROM Table AS T

I’m not sure if that was the only part you were having trouble with. If you have any other questions about the SQL I’d be glad to try to explain…

By: Patrick

Patrick — Tue, 05 Aug 2008 04:20:58 +0000

can you explain the Sql a little more. Im not sure if t is a table or..

Sorry, new a the complex selects

By: Jay

Jay — Thu, 03 Apr 2008 23:03:42 +0000

I appreciate your frustration, but the real problem is that MySQL does not have a simple and elegant solution to this problem; that's probably why you haven't found one you like yet. That said, I did mention the caveat that you need to have sequential IDs for this to work as intended. For our data sets, that is not a problem, but it might be for yours. That said, it should not be possible for this to return a null set; that's why it's x ON T.ID >= x.ID instead of x ON T.ID = x.ID There is another bug in my code though: since MySQL auto-increments start at 1 rather than 0, FLOOR(MAX(ID)*RAND()) should be CEIL(MAX(ID)*RAND()) - I have edited the original post to reflect that.