AVOID DEADLOCK AFTER RESTORE

Analysis
--------
Accessing the restored NDB table in an active multi-statement
transaction was resulting in deadlock found error.

MySQL Server needs to discover metadata of NDB table from
data nodes after table is restored from backup. Metadata
discovery happens on the first access to restored table.
Current code mandates this statement to be the first one
in the transaction. This is because discover needs exclusive
metadata lock on the table. Lock upgrade at this point can
lead to MDL deadlock and the code was written at the time
when MDL deadlock detector was not present. In case when
discovery attempted in the statement other than the first
one in transaction ER_LOCK_DEADLOCK error is reported
pessimistically.

Fix:
---
Removed the constraint as any potential deadlock will be
handled by deadlock detector. Also changed code in discover
to keep metadata locks of active transaction.

Same issue was present in table auto repair scenario. Same
fix is added in repair path also.

Scenario:

1. The purge thread takes an undo log record and parses it and forms
   the record to be purged. We have the primary and secondary keys
   to locate the actual records.
2. Using the secondary index key, we search in the secondary index.
   One record is found.
3. Then it is checked if this record can be purged.  The answer is we
   can purge this record.  To determine this we look up the clustered
   index record.  Either there is no corresponding clustered index
   record, or the matching clustered index record is delete marked.
4. Then we check whether the secondary index record is delete marked.
   We find that it is not delete marked.  We report warning in optimized
   build and assert in debug build.

Problem:

In step 3, we report that the record is purgeable even though it is
not delete marked.  This is because of inconsistency between the
following members of purge_node_t structure - found_clust, ref and pcur.

Solution:

In the row_purge_reposition_pcur(), if the persistent cursor restore
fails, then reset the purge_node_t->found_clust member.  This will
keep the members of purge_node_t structure in a consistent state.

rb#8813 approved by Marko.

As part of the fix find_files() prototype has been modified and
mysql-cluster uses find_files() function. Hence modified find_files() call
in ha_ndbcluster_binlog.cc file to make mysql-cluster build successful.

                 HOST WHEN IT CONTAINS WILDCARD

Description :- Incorrect access privileges are provided to a
user due to wrong sorting of users when wildcard characters
is present in the hostname.

Analysis :- Function "get_sorts()" is used to sort the
strings of user name, hostname, database name. It is used
to arrange the users in the access privilege matching order.
When a user connects, it checks in the sorted user access
privilege list and finds a corresponding matching entry for
the user. Algorithm used in "get_sort()" sorts the strings
inappropriately. As a result, when a user connects to the
server, it is mapped to incorrect user access privileges.
Algorithm used in "get_sort()" counts the number of
characters before the first occurence of any one of the
wildcard characters (single-wildcard character '_' or
multi-wildcard character '%') and sorts in that order.
As a result of inconnect sorting it treats hostname "%" and
"%.mysql.com" as equally-specific values and therefore
the order is indeterminate.

Fix:- The "get_sort()" algorithm has been modified to treat
"%" seperately. Now "get_sort()" returns a number which, if
sorted in descending order, puts strings in the following
order:-
* strings with no wildcards
* strings containg wildcards and non-wildcard characters
* single muilt-wildcard character('%')
* empty string.

Description: On an example MySQL instance with 28k empty
InnoDB tables, a specific query to information_schema.tables
and information_schema.columns leads to memory consumption
over 38GB RSS.

Analysis: In get_all_tables() call, we fill the I_S tables
from frm files and storage engine. As part of that process
we call make_table_name_list() and allocate memory for all
the 28k frm file names in the THD mem_root through
make_lex_string_root(). Since it has been called around
28k * 28k times there is a huge memory getting hogged in
THD mem_root. This causes the RSS to grow to 38GB.

Fix: As part of fix we are creating a temporary mem_root
in get_all_tables and passing it to fill_fiels(). There we
replace the THD mem_root with the temporary mem_root and
allocates the file names in temporary mem_root and frees
it once we fill the I_S tables in get_all_tables and
re-assign the original mem_root back to THD mem_root.

Note: Checked the massif out put with the fix now the memory growth is just around 580MB at peak.

Restrict when user table hashes can be viewed. Require SUPER privileges.

Description:- There is a possibility of negative array index
write associated with the function "terminal_writec()". This
is due to the assumption that there is a possibility of
getting -1 return value from the function call
"ct_visual_char()".

Analysis:- The function "terminal_writec()" is called only
from "em_delete_or_list()" and "vi_list_or_eof()" and both
these functions deal with the "^D" (ctrl+D) signal. So the
"size_t len" and "Char c" passed to "ct_visual_char()" (when
called from "terminal_writec()") is always 8 (macro
VISUAL_WIDTH_MAX is passed whose value is 8) and 4 (ASCII
value for "^D"/"ctrl+D") respectively.
Since the value of "c" is 4, "ct_chr_class()" returns -1
(macro CHTYPE_ASCIICTL is associated with -1 value). And
since value of "len" is 8, "ct_visual_char()" will always
return 2 when it is called from "terminal_writec()".
So there is no possible case so that we encounter a negative
array index write in "terminal_writec()". But since there is
a rare posibility of using "terminal_writec()" in future
enhancements, it is good handle the error case as well.

Fix:- A condition is added in "terminal_writec()" to check
whether "ct_visual_char()" is returning -1 or not. If the
return value is -1, then value 0 is returned to its calling
function "em_delete_or_list()" or "vi_list_or_eof()", which
in turn will return CC_ERROR.

NOTE:- No testcase is added since currently there is no
possible scenario to encounter this error case.

post push change: fixing valgrind failures

post push change: missed the change in mysql-5.5
(Fixing compiler warning/error)

Description:
Can't build mysql-5.5 latest source with openssl 0.9.8e.

Analysis:
Older OpenSSL versions(prior to openssl 1.0) doesn't have 'SSL_OP_NO_COMPRESSION' defined.
Hence the build is failing with SSL_OP_NO_COMPRESSION undeclared.

Fix:
Added a conditonal compilation for 'SSL_OP_NO_COMPRESSION'.
i.e if 'SSL_OP_NO_COMPRESSION' is defined then have the SSL_set_options call for OpenSSL 1.0 versions.
Have sk_SSL_COMP_zero() call for OpenSSL 0.9.8 version

One can not see in PB2 test logs which unit tests have been run
and passed.

This patchs adds an option --unit-tests-report to mtr which
include the ctest report in mtr output.  It will also turn on unit
testing if not explicitly turned off with --no-unit-tests or
equivalent.

In manual runs one can always look in the ctest.log file in mtr
vardir.

--unit-tests are replaced with --unit-tests-report in files under
mysql-test/collections/ to activate report in PB2.

Fix typo: s/COPY_ONLY/COPYONLY/g

(cherry picked from commit 034046b4dfe3e6f83e0cf73310884334e0507f06)

Conflicts:
	cmake/make_dist.cmake.in

               MYISAM TABLE CAUSES THE SERVER TO CRASH

Backport to mysql-5.1

Embedded server does not support restarting and needs to skip the test.

if XA PREPARE transactions hold explicit locks.

innobase_shutdown_for_mysql(): Call trx_sys_close() before lock_sys_close()
(and dict_close()) so that trx_free_prepared() will see all locks intact.

RB: 8561
Reviewed-by: Vasil Dimov <vasil.dimov@oracle.com>

FILL_VARIABLES

Prevent mutexes used in SHOW VARIABLES from being locked twice.

Posty push fix for test case

Backporting the patch to 5.1 and 5.5

PROBLEM

Create time is calculated as last status change time of .frm file.
The first problem was that innodb was passing file name as
"table_name#po#p0.frm" to the stat() call which calculates the create time.
Since there is no frm file with this name create_time will be stored as NULL.
The second problem is ha_partition::info() updates stats for create time
when HA_STATUS_CONST flag was set ,where as innodb calculates this statistic
when HA_STATUS_TIME is set,which causes create_time to be set as NULL.

Fix
Pass proper .frm name to stat() call and calculate create time when
HA_STATUS_CONST flag is set.

Description: SSL tests are failing in mysql-5.1 pb2

Analysis: The SSL certificates are ended by jan 2015.
Hence the SSL tests are failing.

Fix: We have generated new certificates with SHA1 algorithm.

Backport from mysql-5.5 to mysql-5.1

Bug# 19699237: UNINITIALIZED VARIABLE IN
               ITEM_FIELD::STR_RESULT LEADS TO INCORRECT
               BEHAVIOR

ISSUE:
------
When the following conditions are satisfied in a query, a
server crash occurs:
a) Two rows are compared using a NULL-safe equal-to operator.
b) Each of these rows belong to different charsets.

SOLUTION:
---------
When one charset is converted to another for comparision,
the constructor of "Item_func_conv_charset" is called.
This will attempt to use the Item_cache if the string is a
constant. This check succeeds because the "used_table_map"
of the Item_cache class is never set to the correct value.
Since it is mistakenly assumed to be a constant, it tries
to fetch the relevant null value related fields which are
yet to be initialized. This results in valgrind issues
and wrong results.

The fix is to update the "used_table_map" of "Item_cache".
This will allow "Item_func_conv_charset" to realise that
this is not a constant.

Problem: UDF doesn't handle the arguments properly when they
         are of string type due to a misplaced break.
         The length of arguments is also not set properly
         when the argument is NULL.

Solution: Fixed the code by putting the break at right place
          and setting the argument length to zero when the
          argument is NULL.

- Updated emdedded to embedded

			TO USE A SECOND WATCH PAGE PER INSTANCE

Description:
	BUF_POOL_WATCH_SIZE is also initialized to number of purge threads.
so BUF_POOL_WATCH_SIZE will never be lesser than number of purge threads.
From the code, there is no scope for purge thread to skip buf_pool_watch_unset.
So there can be at most one buffer pool watch active per purge thread.
In other words, there is no chance for purge thread instance to hold a watch
when setting another watch.

Solution:
	Adding code comments to clarify the issue.
Reviewed-by: Marko Mäkelä <marko.makela@oracle.com>
Approved via Bug page.

Backport from mysql-5.5 to mysql-5.1

Bug#19880368 : GROUP_CONCAT CRASHES AFTER DUMP_LEAF_KEY

Problem:
find_order_by_list does not update the address of order_item
correctly after resolving.

Solution:
Change the ref_by address for a order_by field if its
SUM_FUNC_ITEM to the address of the field present in
all_fields.

Backport from mysql-5.5 to mysql-5.1

Bug #19612819 :  FILESORT: ASSERTION FAILED: POS->FIELD != 0 || POS->ITEM != 0

Problem:
While getting the temp table field for a REF_ITEM
make_sortorder is using the real_item. As a result
server fails later with an assert.

Solution:
Do not use real_item to get the temp table field.
Instead use the REF_ITEM itself as temp table fields
are created for REF_ITEM not the real_item.