Fix for MDEV-5629: Failing assertion: state == TRX_STATE_NOT_STARTED on concurrent CREATE OR REPLACE and transactional UPDATE

mysql-test/suite/innodb/r/row_lock.result:
  Test case for MDEV-5629
mysql-test/suite/innodb/t/row_lock.test:
  Test case for MDEV-5629
sql/filesort.cc:
  Don't call unlock_row() in case of errors