How fast can you sort data with MySQL ? - MySQL Performance Blog

How fast can you sort data with MySQL ? - MySQL Performance Blog: "set sort_buffer_size=100000;"

I took the same table as I used for MySQL Group by Performance Tests to see how much MySQL can sort 1.000.000 rows, or rather return top 10 rows from sorted result set which is the most typical way sorting is used in practice.

I tested full table scan of the table completes in 0.22 seconds giving us about 4.5 Million of rows/sec. Obviously we can’t get sorted result set faster than that.

I placed temporary sort files on tmpfs (/dev/shm) to avoid disk IO as a variable as my data set fits in memory anyway and decided to experiment with sort_buffer_size variable.

The minimum value for sort_buffer_size is 32K which gives us the following speed:

mysql> select * from gt order by i desc limit 10;
+--------+------------------------------------------+
| i      | c                                        | 
+--------+------------------------------------------+
| 100000 | 635e8e8f8e3b9dc547bbd3deaadb1f297f691729 |
| 100000 | 0a7750a1393e77a2871ecfb39d5032d0b0f7c37c |
| 100000 | 0db0601036fb9d1d5e17631d4d1bed9149675bb3 | 
| 100000 | eb6d2b5ed1897bdd0ff6e22ee1b44814ffb8f912 |
| 100000 | 1bff67cc134e316dad5370de38020bef818ec45c |
|  99999 | 635da2e73d88dbe5f7297253680398e58d32ff65 |
|  99999 | a1feec5f8ee6c6a96723a2a0b57c418bb3ced929 | 
|  99999 | 72b934f76863791f740b96858d5acb6a60459644 |
|  99999 | 855b47aaa25054e77dcc27de5def8de1e265f371 |
|  99999 | 81980bcd9dbaa565f22a93ce1faf9e9d53407f0a |
+--------+------------------------------------------+ 
10 rows in set (0.56 sec)

Not bad ! Even though MySQL does not optimize “get top N sorted rows” very well it takes just 2.5 times longer than full table scan to get the data. And this is with minimum sort_buffer allowed when a lot of sort merge passes are required for sort completion:

mysql> show status  like "sort%";
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Sort_merge_passes | 321   |
| Sort_range        | 0     |
| Sort_rows         | 10    |
| Sort_scan         | 1     |
+-------------------+-------+
4 rows in set (0.00 sec)

As you can see from this show status output MySQL only counts completely sorted rows in Sort_rows variable. In this case 1.000.000 of rows were partially sorted but only 10 rows fetched from the data file and sent and only they are counted. In practice this means Sort_rows may well understate sort activity happening on the system.

Lets now increase sort_buffer_size and see how performance is affected:

set sort_buffer_size=100000;

mysql> select * from gt order by i desc limit 10;
10 rows in set (0.44 sec)

mysql> show status  like "sort%";
+-------------------+-------+
| Variable_name     | Value | 
+-------------------+-------+
| Sort_merge_passes | 104   |
| Sort_range        | 0     |
| Sort_rows         | 10    |
| Sort_scan         | 1     |
+-------------------+-------+
4 rows in set (0.00 sec)

OK raising sort_buffer_size to 100K gives quite expected performance benefit, now we’re just 2 times slower than table scan of the query and considering table size was about 60MB we have 120MB/sec sort speed, while 2.000.000 rows/sec is of course more relevant in this case.

Still a lot of sort merge passes lets go with even higher buffer sizes.

set sort_buffer_size=1000000;

mysql> select * from gt order by i desc limit 10;
10 rows in set (0.70 sec)

mysql> show status  like "sort%";
+-------------------+-------+
| Variable_name     | Value | 
+-------------------+-------+
| Sort_merge_passes | 10    |
| Sort_range        | 0     |
| Sort_rows         | 10    |
| Sort_scan         | 1     |
+-------------------+-------+
4 rows in set (0.00 sec)

set sort_buffer_size=10000000;

mysql> select * from gt order by i desc limit 10;
10 rows in set (1.34 sec)

mysql> show status  like "sort%";
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Sort_merge_passes | 1     |
| Sort_range        | 0     |
| Sort_rows         | 10    |
| Sort_scan         | 1     |
+-------------------+-------+ 
4 rows in set (0.00 sec)

Wait it is not right. We’re increasing sort_buffer_size and number of sort_merge_passes decreases appropriately but it does not help sort speed instead it drops 3 times from 0.44sec to do 1.34sec !

Lets try it even higher to finally get rid of sort merge passes – may be it is sort merge which is inefficient with largesort_buffer_size ?

mysql> set sort_buffer_size=100000000;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from gt order by i desc limit 10;
+--------+------------------------------------------+ 
| i      | c                                        |
+--------+------------------------------------------+
| 100000 | eb6d2b5ed1897bdd0ff6e22ee1b44814ffb8f912 |
| 100000 | 635e8e8f8e3b9dc547bbd3deaadb1f297f691729 | 
| 100000 | 1bff67cc134e316dad5370de38020bef818ec45c |
| 100000 | 0db0601036fb9d1d5e17631d4d1bed9149675bb3 |
| 100000 | 0a7750a1393e77a2871ecfb39d5032d0b0f7c37c |
|  99999 | 41f091f4074717bf80d2b1a788e6a4a122057d11 | 
|  99999 | 049d9591ef0f584deaaf0433c0f3eda8631bdb85 |
|  99999 | 72b934f76863791f740b96858d5acb6a60459644 |
|  99999 | f0a42a16a41b4249da7c31f2d9556f05622a87b4 |
|  99999 | 35de8ae483779e6024c51998eb5b5e69e02eb74c | 
+--------+------------------------------------------+
10 rows in set (1.55 sec)

mysql> show status  like "sort%";
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+ 
| Sort_merge_passes | 0     |
| Sort_range        | 0     |
| Sort_rows         | 10    |
| Sort_scan         | 1     |
+-------------------+-------+
4 rows in set (0.00 sec)

Nope. We finally got rid of sort_merge_passes but our sort performance got even worse !

I decided to experiment a bit further to see what sort_buffer_size is optimal for given platform and given query (I did not test if it is the same for all platforms or data sets) – The optimal sort_buffer_size in this case was 70K-250K which is quite smaller than even default value.

The CPU in question was Pentium 4 having 1024K of cache.

A while ago I already wrote what large buffers are not always better but I never expected optimal buffer to be so small at least in some conditions.

What do we learn from these results:

• Benchmark your application Unfortunately general tuning guidelines can be wrong for your particular case, or generally wrong because they tend to reprint the manual which is often written based on theoretical expectations rather than supported by large amount of testing.
• sort_merge_passes are not that bad. Setting your sort_buffer_size large enough so there is zero sort_merge_passes may not be optimal.
• World is full of surprises I obviously did not expect to get such results, and this is not any exception. Even spending a lot of time optimizing and otherwise working with MySQL I continue to run in results which surprise me. Some are later expected others come from underlying bugs and later fixed.