Bug#41756 "Strange error messages about locks from InnoDB".

In JT_EQ_REF (join_read_key()) access method,
don't try to unlock rows in the handler, unless certain that
a) they were locked
b) they are not used.

Unlocking of rows is done by the logic of the nested join loop,
and is unaware of the possible caching that the access method may
have. This could lead to double unlocking, when a row
was unlocked first after reading into the cache, and then
when taken from cache, as well as to unlocking of rows which
were actually used (but taken from cache).

Delegate part of the unlocking logic to the access method,
and in JT_EQ_REF count how many times a record was actually
used in the join. Unlock it only if it's usage count is 0.

Implemented review comments.


mysql-test/r/bug41756.result:
  Add result file (Bug#41756)
mysql-test/t/bug41756-master.opt:
  Use --innodb-locks-unsafe-for-binlog, as in 5.0 just
  using read_committed isolation is not sufficient to 
  reproduce the bug.
mysql-test/t/bug41756.test:
  Add a test file (Bug#41756)
sql/item_subselect.cc:
  Complete struct READ_RECORD initialization with a new
  member to unlock records.
sql/records.cc:
  Extend READ_RECORD API with a method to unlock read records.
sql/sql_select.cc:
  In JT_EQ_REF (join_read_key()) access method,
  don't try to unlock rows in the handler, unless certain that
  a) they were locked
  b) they are not used.
sql/sql_select.h:
  Add members to TABLE_REF to count TABLE_REF buffer usage count.
sql/structs.h:
  Update declarations.

When a sessione is closed, all temporary tables of the session are automatically 
dropped and are binlogged. But it will be binlogged with wrong database names when
the length of the temporary tables' database names are greater than the 
length of the current database name or the current database is not set.

Query_log_event's db_len is forgot to set when Query_log_event's db is set.
This patch wrote code to set db_len immediately after db has set.

Backport a ndb patch: fix bug with crash during restart, where
a mbyte incorrectly could be skipped, leading to "end of log
wo/ finding gci".

Restore a stub of the removed mysql_odbc_escape_string function
to fix a ABI breakage. The function was intended to be private
and used only by Connector/ODBC, but, unfortunately, it was exported
as part of the ABI. Nonetheless, only a stub is restored as the
original function is inherently broken and shouldn't be used.

This restoration only applies to MySQL 5.0. This will be addressed
differently in later versions -- reworked library versioning.

include/mysql.h:
  Restore mysql_odbc_escape_string prototype.
include/mysql_h.ic:
  Update ABI check.
libmysql/libmysql.c:
  Restore a mysql_odbc_escape_string stub.
libmysql/libmysql.def:
  Restore mysql_odbc_escape_string.
libmysqld/libmysqld.def:
  Restore mysql_odbc_escape_string.

            with temporary tables

There were two problems the test case from this bug was
triggering:

1. JOIN::rollup_init() was supposed to wrap all constant Items
into another object for queries with the WITH ROLLUP modifier
to ensure they are never considered as constants and therefore
are written into temporary tables if the optimizer chooses to
employ them for DISTINCT/GROUP BY handling.

However, JOIN::rollup_init() was called before
make_join_statistics(), so Items corresponding to fields in
const tables could not be handled as intended, which was
causing all kinds of problems later in the query execution. In
particular, create_tmp_table() assumed all constant items
except "hidden" ones to be removed earlier by remove_const()
which led to improperly initialized Field objects for the
temporary table being created. This is what was causing crashes
and valgrind errors in storage engines.

2. Even when the above problem had been fixed, the query from
the test case produced incorrect results due to some
DISTINCT/GROUP BY optimizations being performed by the
optimizer that are inapplicable in the WITH ROLLUP case.

Fixed by disabling inapplicable DISTINCT/GROUP BY optimizations
when the WITH ROLLUP modifier is present, and splitting the
const-wrapping part of JOIN::rollup_init() into a separate
method which is now invoked after make_join_statistics() when
the const tables are already known.

mysql-test/r/olap.result:
  Added a test case for bug #48131.
mysql-test/t/olap.test:
  Added a test case for bug #48131.
sql/sql_select.cc:
  1. Disabled inapplicable DISTINCT/GROUP BY optimizations when
  the WITH ROLLUP modifier is present.
  2. Split the const-wrapping part of JOIN::rollup_init() into a
  separate method.
sql/sql_select.h:
  Added rollup_process_const_fields() declaration.

  subquery returning multiple rows

Error handling was missing when handling subqueires in WHERE 
and when assigning a SELECT result to a @variable.
This caused crash(es). 

Fixed by adding error handling code to both the WHERE 
condition evaluation and to assignment to an @variable.

having clause...

The fix for bug 46184 was not very complete. It was not covering
views using temporary tables and multiple tables in a FROM clause.
Fixed by reverting the fix for 46184 and making a more general
check that is checking at the right execution stage and for all
of the non-supported cases.
Now PROCEDURE ANALYZE on non-top level SELECT is also forbidden.
Updated the analyse.test and subselect.test accordingly.

Queries with nested outer joins may lead to crashes or 
bad results because an internal data structure is not handled
correctly.
The optimizer uses bitmaps of nested JOINs to determine
if certain table can be placed at a certain place in the
JOIN order.
It does maintain a bitmap describing in which JOINs 
last placed table is nested.
When it puts a table it makes sure the bit of every JOIN that
contains the table in question is set (because JOINs can be nested).
It does that by recursively setting the bit for the next enclosing
JOIN when this is the first table in the JOIN and recursively 
resetting the bit if it's the last table in the JOIN.
When it removes a table from the join order it should do the
opposite : recursively unset the bit if it's the only remaining 
table in this join and and recursively set the bit if it's removing
the last table of a JOIN.
There was an error in how the bits was set for the upper levels :
when removing a table it was setting the bit for all the enclosing 
nested JOINs even if there were more tables left in the current JOIN
(which practically means that the upper nested JOINs were not affected).
Fixed by stopping the recursion at the relevant level.

mysql-test/r/join.result:
  Bug #42116: test case
mysql-test/t/join.test:
  Bug #42116: test case
sql/sql_select.cc:
  Bug #41116: don't go up and set the bits if more tables in
  at the current JOIN level

test result fix

mysql-test/suite/funcs_1/r/is_columns_mysql.result:
  test result fix
mysql-test/suite/funcs_1/r/is_statistics.result:
  test result fix

BUG#41597 - After rename of user, there are additional grants
            when grants are reapplied.

Fixed build failure on Windows. Added missing cast.

sql/sql_acl.cc:
  Fixed build failure on Windows. Added missing cast.

Problem 1:
column_priv_hash uses utf8_general_ci collation
for the key comparison. The key consists of user name,
db name and table name. Thus user with privileges on table t1
is able to perform the same operation on T1
(the similar situation with user name & db name, see acl_cache).
So collation which is used for column_priv_hash and acl_cache
should be case sensitive.
The fix:
replace system_charset_info with my_charset_utf8_bin for
column_priv_hash and acl_cache
Problem 2:
The same situation with proc_priv_hash, func_priv_hash,
the only difference is that Routine name is case insensitive.
So the fix is to use my_charset_utf8_bin for
proc_priv_hash & func_priv_hash and convert routine name into lower
case before writing the element into the hash and
before looking up the key.
Additional fix: mysql.procs_priv Routine_name field collation
is changed to utf8_general_ci.
It's necessary for REVOKE command
(to find a field by routine hash element values).
Note: 
It's safe for lower-case-table-names mode too because
db name & table name are converted into lower case
(see GRANT_NAME::GRANT_NAME).


mysql-test/include/have_case_insensitive_fs.inc:
  test case
mysql-test/r/case_insensitive_fs.require:
  test case
mysql-test/r/grant_lowercase_fs.result:
  test result
mysql-test/r/lowercase_fs_off.result:
  test result
mysql-test/r/ps_grant.result:
  test result
mysql-test/r/system_mysql_db.result:
  changed Routine_name field collation to case insensitive
mysql-test/t/grant_lowercase_fs.test:
  test case
mysql-test/t/lowercase_fs_off.test:
  test case
scripts/mysql_system_tables.sql:
  changed Routine_name field collation to case insensitive
scripts/mysql_system_tables_fix.sql:
  changed Routine_name field collation to case insensitive
sql/sql_acl.cc:
  Problem 1:
  column_priv_hash uses utf8_general_ci collation
  for the key comparison. The key consists of user name,
  db name and table name. Thus user with privileges on table t1
  is able to perform the same operation on T1
  (the similar situation with user name & db name, see acl_cache).
  So collation which is used for column_priv_hash and acl_cache
  should be case sensitive.
  The fix:
  replace system_charset_info with my_charset_utf8_bin for
  column_priv_hash and acl_cache
  Problem 2:
  The same situation with proc_priv_hash, func_priv_hash,
  the only difference is that Routine name is case insensitive.
  So the fix is to use my_charset_utf8_bin for
  proc_priv_hash & func_priv_hash and convert routine name into lower
  case before writing the element into the hash and
  before looking up the key.
  Additional fix: mysql.procs_priv Routine_name field collation
  is changed to utf8_general_ci.
  It's necessary for REVOKE command
  (to find a field by routine hash element values).
  Note: 
  It's safe for lower-case-table-names mode too because
  db name & table name are converted into lower case
  (see GRANT_NAME::GRANT_NAME).

      
If the first argument to GeomFromWKB function is a geometry
field then the function just returns its value.
However in doing so it's not preserving first argument's 
null_value flag and this causes unexpected null value to
be returned to the calling function.
      
Fixed by updating the null_value of the GeomFromWKB function
in such cases (and all other cases that return a NULL e.g.
because of not enough memory for the return buffer).

Problem: involving a spatial index for "non-spatial" queries
(that don't containt MBRXXX() functions) may lead to failed assert.

Fix: don't use spatial indexes in such cases.


mysql-test/r/gis-rtree.result:
  Fix for bug#48258: Assertion failed when using a spatial index
    - test result.
mysql-test/t/gis-rtree.test:
  Fix for bug#48258: Assertion failed when using a spatial index
    - test case.
sql/opt_range.cc:
  Fix for bug#48258: Assertion failed when using a spatial index
    - allow only spatial functions (MBRXXX) for itMBR keyparts.

line 138 when forcing a spatial index

Problem: "Spatial indexes can be involved in the search 
for queries that use a function such as MBRContains() 
or MBRWithin() in the WHERE clause".
Using spatial indexes for JOINs with =, <=> etc.
predicates is incorrect.

Fix: disable spatial indexes for such queries.


mysql-test/r/select.result:
  Fix for bug#47019: Assertion failed: 0, file .\rt_mbr.c, 
  line 138 when forcing a spatial index
    - test result.
mysql-test/t/select.test:
  Fix for bug#47019: Assertion failed: 0, file .\rt_mbr.c, 
  line 138 when forcing a spatial index
    - test case.
sql/sql_select.cc:
  Fix for bug#47019: Assertion failed: 0, file .\rt_mbr.c, 
  line 138 when forcing a spatial index
    - disable spatial indexes for queries which use 
  non-spatial conditions (e.g. NATURAL JOINs).

Removed the verify callback, as it's not needed to verify even self
signed certificates and is a security problem.

                     grants are reapplied.


After renaming a user and trying to re-apply grants results in additional
grants.

This is because we use username as part of the key for GRANT_TABLE structure.
When the user is renamed, we only change the username stored and the hash key
still contains the old user name and this results in the extra privileges

Fixed by rebuilding the hash key and updating the column_priv_hash structure
when the user is renamed

mysql-test/r/grant3.result:
  Bug #41597 - After rename of user, there are additional grants when 
               grants are reapplied.
  
  Testcase for BUG#41597
mysql-test/t/grant3.test:
  Bug #41597 - After rename of user, there are additional grants when 
               grants are reapplied.
  
  Testcase for BUG#41597
sql/sql_acl.cc:
  Bug #41597 - After rename of user, there are additional grants when 
               grants are reapplied.
  
  Fixed handle_grant_struct() to update the hash key when the user is renamed.
  Added to set_user_details() method to GRANT_NAME class

If a thread is killed in the server, we throw "shutdown" only if one is actually in
progress; otherwise, we throw "query interrupted".

Control-C in the mysql command-line client is "incremental" now.
First Control-C sends KILL QUERY (when connected to 5.0+ server, otherwise, see next)
Next  Control-C sends KILL CONNECTION
Next  Control-C aborts client.

As the first two steps only pertain to an existing query,
Control-C will abort the client right away if no query is running.

client will give more detailed/consistent feedback on Control-C now.


client/mysql.cc:
  Extends Control-C handling; enhances up feedback to user.
  
  On 5.0+ servers, we try to be nice and send KILL QUERY first
  if Control-C is pressed in the command-line client, but if
  that doesn't work, we now give the user the opportunity to
  send KILL CONNECTION with another Control-C (and to kill the
  client with another Control-C if that somehow doesn't work
  either).
mysql-test/t/flush_read_lock_kill.test:
  we're getting correct "thread killed" rather than
  "in shutdown" error now
mysql-test/t/kill.test:
  we're getting correct "thread killed" rather than
  "in shutdown" error now
mysql-test/t/rpl000001.test:
  we're getting correct "thread killed" rather than
  "in shutdown" error now
mysql-test/t/rpl_error_ignored_table.test:
  we're getting correct "thread killed" rather than
  "in shutdown" error now
sql/records.cc:
  make error messages on KILL uniform for rr_*()
  by folding that handling into rr_handle_error()
sql/sql_class.h:
  Only throw "shutdown" when we have one flagged as being in progress;
  otherwise, throw "query interrupted" as it's likely to be "KILL CONNECTION"
  or related.

Use "#ifdef", not plain "#if".

Put variable declaration at the beginning of a block

The bug is a compilation issue:
Function "find_key_block()" had thread operations
which were not guarded by "#if THREAD", add that now.

mysys/mf_keycache.c:
  Without thread-support, we are in a single-threaded world,
  so there is no parallelism, and the keycache cannot be
  within a resize operation while this request is being processed.

can lead to bad memory access

Problem: Field_bit is the only field which returns INT_RESULT
and doesn't have unsigned flag. As it's not a descendant of the 
Field_num, so using ((Field_num *) field_bit)->unsigned_flag may lead
to unpredictable results.

Fix: check the field type before casting.


mysql-test/r/type_bit.result:
  Fix for bug #42803: Field_bit does not have unsigned_flag field,
  can lead to bad memory access
    - test result.
mysql-test/t/type_bit.test:
  Fix for bug #42803: Field_bit does not have unsigned_flag field,
  can lead to bad memory access
    - test case.
sql/opt_range.cc:
  Fix for bug #42803: Field_bit does not have unsigned_flag field,
  can lead to bad memory access
    - don't cast to (Field_num *) Field_bit, as it's not a Field_num
  descendant and is always unsigned by nature.