Backport of Bug#13581962

mysql-test/r/cast.result:
  Added test result for Bug#13581962,Bug#14096619
mysql-test/t/cast.test:
  Added test case for Bug#13581962,Bug#14096619
sql/item_func.h:
  limit max length by MY_INT64_NUM_DECIMAL_DIGITS