The out of memory error was thrown when the sort buffer size were too small.
This led to a user confusion.

Now filesort throws the error message about sort buffer being too small.