Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE

            causes incorrect duplicate entries

Keys for BTREE indexes on ENUM and SET columns of MEMORY tables
with character set UTF8 were computed incorrectly. Many
different column values got the same key value.

Apart of possible performance problems, it made unique indexes
of this type unusable because it rejected many different
values as duplicates.

The problem was that multibyte character detection was tried
on the internal numeric column value. Many values were not
identified as characters. Their key value became blank filled.

Thanks to Alexander Barkov and Ramil Kalimullin for the patch,
which sets the character set of ENUM and SET key segments to
the pseudo binary character set.

mysql-test/r/heap_btree.result

@@ -280,4 +280,16 @@ a
 1
 1
 drop table t1;
+CREATE TABLE t1 (
+c1 ENUM('1', '2'),
+UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
+CREATE TABLE t1 (
+c1 SET('1', '2'),
+UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
 End of 4.1 tests

mysql-test/t/heap_btree.test

@@ -182,4 +182,21 @@ delete from t1 where a >= 2;
 select a from t1 order by a;
 drop table t1;
 
+#
+# Bug#24985 - UTF8 ENUM primary key on MEMORY using BTREE
+#             causes incorrect duplicate entries
+#
+CREATE TABLE t1 (
+  c1 ENUM('1', '2'),
+  UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
+CREATE TABLE t1 (
+  c1 SET('1', '2'),
+  UNIQUE USING BTREE(c1)
+) ENGINE= MEMORY DEFAULT CHARSET= utf8;
+INSERT INTO t1 VALUES('1'), ('2');
+DROP TABLE t1;
+
 --echo End of 4.1 tests

sql/ha_heap.cc

@@ -549,6 +549,9 @@ int ha_heap::create(const char *name, TABLE *table_arg,
       seg->start=   (uint) key_part->offset;
       seg->length=  (uint) key_part->length;
       seg->flag =   0;
+      if (field->flags & (ENUM_FLAG | SET_FLAG))
+        seg->charset= &my_charset_bin;
+      else
         seg->charset= field->charset();
       if (field->null_ptr)
       {