in the case of the overflow in the decimal->integer conversion
we didn't return the proper boundary value, but just the result
of the conversion we calculated on the moment of the error


mysql-test/r/bigint.result:
  bug #8663 cant use bigint unsigned as input to cast
  test result fixed
mysql-test/t/bigint.test:
  bug #8663 cant use bigint unsigned as input to cast
  test case
strings/decimal.c:
  bug #8663 cant use bigint unsigned as input to cast
  decimal->int conversion fixed to return proper boundary value
  in the case of overflow