This patch removes most remaining delays due to uninteruptible sleep()
during shutdown, as found using PMP. This makes standard test run very
close in speed to with --loose-innodb-fast-shutdown=2, and greatly
speeds up running the test suite.

sql/mysqld.cc:
  Poll for threads to die every 20 msec during shutdown, rather than force
  a wait for 2 full seconds should one thread be a little slow to exit
  (this was seen occasionally in mysql-test-run).
storage/xtradb/include/srv0srv.h:
  Rename the event, as we now use it to wakeup more threads during shutdown.
storage/xtradb/log/log0log.c:
  Rename the event, as we now use it to wakeup more threads during shutdown.
storage/xtradb/srv/srv0srv.c:
  Replace some hardcoded sleep()s with os_wait_event_time() that can be
  interrupted early during server shutdown to avoid unnecessary delays.

- Fixed bug in pagecache_delete_internal() when deleting block that was flushed by another thread (fixed bug when block->next_used was unexpectedly null)
Fixed some using uninitialized memory warnings found by valgrind. 

mysql-test/t/information_schema_all_engines-master.opt:
  Added options to make slow test run faster
sql/sp.cc:
  Fixed valgrind warning.
sql/sql_show.cc:
  Fixed valgrind warning.
storage/maria/ma_bitmap.c:
  Fixed wrong call parameter to pagecache_unlock_by_link() that caused assert crash in pagecache
storage/maria/ma_pagecache.c:
  Fixed possible error in pagecache_wait_lock() when hash_link was not set. (We never hit this issue, but could be possible when two threads updates the same page.
  Fixed bug in pagecache_delete_internal() when deleting block that was flushed by another thread (fixed bug when block->next_used was unexpectedly null)
  Cleanup: moved pagecache_pthread_mutex_unlock() over comments and asserts to be just before pagecache_fwrite()

Added option BACKUP_ALL to mysqld --myisam-recover to also get a backup of the index file before it's repaired.
Removed wrong call to translog_buffer_unlock() that caused 'unlocking not locked mutex' failure in Aria log handler.

extra/replace.c:
  Updated call to my_redel()
include/maria.h:
  Updated prototype for maria_change_to_newfile
include/my_sys.h:
  Updated prototypes
  Added my_create_backup_name
include/myisam.h:
  Updated prototypes
include/myisamchk.h:
  Added 'backup_time' to st_handler_check_param to be able to generate same name for backuped data and index file
mysys/my_redel.c:
  Added time_t option to my_redel() to be able to generate same backup extensions for many files
sql/handler.cc:
  Added start_time to st_ha_check_opt
sql/handler.h:
  Added start_time to HA_CHECK_OPT
sql/mysqld.cc:
  Added option BACKUP_ALL to --myisam-recover
storage/maria/ha_maria.cc:
  Remember start time for repair
storage/maria/ma_check.c:
  Use remembered start time for backup file names
  Removed some dead code
storage/maria/ma_loghandler.c:
  Removed wrong call to translog_buffer_unlock() that caused 'unlocking not locked mutex' failure in log handler.
storage/maria/maria_chk.c:
  Removed dead code (O_NEW_INDEX was never set)
  Report if table was 'crashed on repair'
storage/maria/maria_pack.c:
  Updated parameters to my_redel()
storage/myisam/ha_myisam.cc:
  Added recover option BACKUP_ALL to get a backup of the index file before it's repaired.
  Print information to log if we make a backup of data as part of repair.
storage/myisam/ha_myisam.h:
  Added HA_RECOVER_FULL_BACKUP
storage/myisam/mi_check.c:
  Use remembered start time for backup file names
  Added mi_make_backup_of_index()
storage/myisam/myisamchk.c:
  Removed dead code (O_NEW_INDEX was never set)
  Report if table was 'crashed on repair'
storage/myisam/myisampack.c:
  Updated call to my_redel()

This fixes a bug that gave ER_FILE_TOO_SHORT error when scanning Aria tables.

storage/maria/ma_blockrec.c:
  Changed code so that we update share->state.state.data_file_length after page cache write is done.
  When scanning pages, stop when you are at 'data_file_length'.
  (We can't trust the bitmap as there may be reserved pages in the bitmap that are not yet written).
  This fixes a bug that gave ER_FILE_TOO_SHORT error when scanning Aria tables.
storage/maria/maria_def.h:
  Updated struct for stopping scanning at end of file

Implement os_event_wait_time() for POSIX systems.

In the purge thread, use os_event_wait_time() when sleeping rather than sleep,
and signal the event when server shuts down, so we do not need to wait for
upto 10 seconds until the purge thread wakes up.

Also fix bug that warnings that were pushed after we call set_ok_status() were
not included in the waning count sent to the client in the result packet.

Also in mysqltest, in recursive die() invocation at least print the message
before aborting.

client/mysqltest.cc:
  If we detect recursive die(), at least print the message before aborting.
mysql-test/r/warnings_debug.result:
  Test case.
mysql-test/t/warnings_debug.test:
  Test case.
sql/handler.cc:
  Force generation of a warning with specific debug option, for testing.
sql/protocol.cc:
  Fix wrong DBUG_ENTER
sql/sql_class.h:
  Add method to count warnings pushed after set_ok_status() is called.
sql/sql_error.cc:
  Also count warnings pushed after set_ok_status() is called.
storage/xtradb/include/os0sync.h:
  Implement working os_sync_wait_time() for POSIX.
storage/xtradb/include/srv0srv.h:
  Make the purge thread wait for an event when sleeping, so we can signal it to
  wakeup immediately at shutdown.
storage/xtradb/log/log0log.c:
  Make the purge thread wait for an event when sleeping, so we can signal it to
  wakeup immediately at shutdown.
storage/xtradb/os/os0sync.c:
  Implement working os_sync_wait_time() for POSIX.
storage/xtradb/srv/srv0srv.c:
  Make the purge thread wait for an event when sleeping, so we can signal it to
  wakeup immediately at shutdown.

Fixed a bug in Aria when two threads was inserting into the same table and row page and one thread did an abort becasue of duplicate key.



mysys/thr_lock.c:
  Fixed valgrind warning
sql/sql_base.cc:
  Remove not used variable
storage/maria/ma_bitmap.c:
  Added ma_bitmap_lock() & ma_bitmap_unlock() to protect against two threads using the bitmap at the same time.
  More DBUG_PRINT()
storage/maria/ma_blockrec.c:
  Fixed a bug in Aria when two threads was inserting into the same table and row page and one thread did an abort becasue of duplicate key.
  Fix was that we block other threads to modify the bitmap while we are removing the row with a duplicate key.
storage/maria/ma_blockrec.h:
  Added ma_bitmap_lock() & ma_bitmap_unlock() to protect against two threads using the bitmap at the same time.
storage/maria/maria_def.h:
  Changed flush_all_requested to be a counter.
storage/myisam/mi_locking.c:
  Fixed compiler error on windows (typo).

Added locking of lock mutex when updating status in external_unlock() for Aria and MyISAM tables.
Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards. (Allows one to run --debug with mysql-test-run scripts that uses @debug)
Fixed several core dump, deadlock and crashed table bugs in handling of LOCK TABLE with MERGE tables:
- Added priority of locks to avoid crashes with MERGE tables.
- Added thr_lock_merge() to allow one to merge two results of thr_lock().
Fixed 'not found row' bug in REPLACE with Aria tables.
Mark MyISAM tables that are part of MERGE with HA_OPEN_MERGE_TABLE and set the locks to have priority THR_LOCK_MERGE_PRIV.
- By sorting MERGE tables last in thr_multi_unlock() it's safer to release and relock them many times (can happen when TRIGGERS are created)
Avoid printing (null) in debug file (to easier find out wrong NULL pointer usage with %s).



client/mysql.cc:
  Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
client/mysqltest.cc:
  Don't send NULL to fn_format(). (Can cause crash on Solaris when using --debug)
dbug/dbug.c:
  DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards.
include/my_base.h:
  Added flag to signal if one opens a MERGE table.
  Added extra() command to signal that one is not part of a MERGE table anymore.
include/thr_lock.h:
  Added priority for locks (needed to fix bug in thr_lock when using MERGE tables)
  Added option to thr_unlock() if get_status() should be called.
  Added prototype for thr_merge_locks().
mysql-test/mysql-test-run.pl:
  Ignore crashed table warnings for tables named 'crashed'.
mysql-test/r/merge.result:
  Renamed triggers to make debugging easier.
  Added some CHECK TABLES to catch errors earlier.
  Additional tests.
mysql-test/r/merge_debug.result:
  Test of error handling when reopening MERGE tables.
mysql-test/r/udf_query_cache.result:
  Added missing flush status
mysql-test/suite/parts/r/partition_repair_myisam.result:
  Update results
mysql-test/t/merge.test:
  Renamed triggers to make debugging easier.
  Added some CHECK TABLES to catch errors earlier.
  Additional tests.
mysql-test/t/merge_debug.test:
  Test of error handling when reopening MERGE tables.
mysql-test/t/udf_query_cache.test:
  Added missing flush status
mysys/my_getopt.c:
  Removed not used variable
mysys/my_symlink2.c:
  Changed (null) to (NULL) to make it easier to find NULL arguments to DBUG_PRINT() functions.
  (On linux, NULL to sprintf is printed 'null')
mysys/thr_lock.c:
  Added priority of locks to avoid crashes with MERGE tables.
  Added thr_lock_merge() to allow one to merge two results of thr_lock().
  - This is needed for MyISAM as all locked table must share the same status. If not, you will not see newly inserted rows in other instances of the table.
  If calling thr_unlock() with THR_UNLOCK_UPDATE_STATUS, call update_status() and restore_status() for the locks. This is needed in some rare cases where we call thr_unlock() followed by thr_lock() without calling external_unlock/external_lock in between.
  Simplify loop in thr_multi_lock().
  Added 'start_trans', which is called at end of thr_multi_lock() when all locks are taken.
  - This was needed by Aria to ensure that transaction is started when we got all locks, not at get_status(). Without this, some rows could not be visible when we lock two tables at the same time, causing REPLACE using two tables to fail unexpectedly.
sql/handler.cc:
  Add an assert() in handler::print_error() for "impossible errors" (like table is crashed) when --debug-assert-if-crashed-table is used.
sql/lock.cc:
  Simplify mysql_lock_tables() code if get_lock_data() returns 0 locks.
  Added new parameter to thr_multi_unlock()
  In mysql_unlock_read_tables(), call first externa_unlock(), then thr_multi_unlock();  This is same order as we do in mysql_unlock_tables().
  Don't abort locks in mysql_lock_abort() for merged tables when a MERGE table is deleted; Would cause a spin lock.
  Added call to thr_merge_locks() in mysql_lock_merge() to ensure consistency in thr_locks().
  - New locks of same type and table is stored after the old lock to ensure that we get the status from the original lock.
sql/mysql_priv.h:
  Added debug_assert_if_crashed_table
sql/mysqld.cc:
  Added --debug-assert-if-crashed-table
sql/parse_file.cc:
  Don't print '(null)' in DBUG_PRINT of no dir given
sql/set_var.cc:
  Increase default size of buffer for @debug variable.
sql/sql_base.cc:
  In case of error from reopen_table() in reopen_tables(), call unlock_open_table() and restart loop.
  - This fixed bug when we twice deleted same table from open_cache.
  Don't take name lock for already name locked table in open_unireg_entry().
  - Fixed bug when doing repair in reopen_table().
  - In detach_merge_children(), always detach if 'clear_refs' is given. We can't trust parent->children_attached as this function can be called twice, first time with clear_refs set to 0.
sql/sql_class.cc:
  Changed printing of (null) to "" in set_thd_proc_info()
sql/sql_parse.cc:
  Added DBUG
sql/sql_trigger.cc:
  Don't call unlink_open_table() if reopen_table() fails as the table may already be freed.
storage/maria/ma_bitmap.c:
  Fixed DBUG_ASSERT() in allocate_tail()
storage/maria/ma_blockrec.c:
  Fixed wrong calculation of row length for very small rows in undo_row_update().
  - Fixes ASSERT() when doing undo.
storage/maria/ma_blockrec.h:
  Added _ma_block_start_trans() and _ma_block_start_trans_no_versioning()
storage/maria/ma_locking.c:
  Call _ma_update_status_with_lock() when releasing write locks.
  - Fixes potential problem with updating status without the proper lock.
storage/maria/ma_open.c:
  Changed to use start_trans() instead of get_status() to ensure that we see all rows in all locked tables when we got the locks.
  - Fixed 'not found row' bug in REPLACE with Aria tables.
storage/maria/ma_state.c:
  Added _ma_update_status_with_lock() and _ma_block_start_trans().
  This is to ensure that we see all rows in all locked tables when we got the locks.
storage/maria/ma_state.h:
  Added _ma_update_status_with_lock()
storage/maria/ma_write.c:
  More DBUG_PRINT
storage/myisam/mi_check.c:
  Fixed error message
storage/myisam/mi_extra.c:
  Added HA_EXTRA_DETACH_CHILD:
  - Detach MyISAM table to not be part of MERGE table (remove flag & lock priority).
storage/myisam/mi_locking.c:
  Call mi_update_status_with_lock() when releasing write locks.
  - Fixes potential problem with updating status without the proper lock.
  Change to use new HA_OPEN_MERGE_TABLE flag to test if MERGE table.
  Added mi_fix_status(), called by thr_merge().
storage/myisam/mi_open.c:
  Added marker if part of MERGE table.
  Call mi_fix_status() in thr_lock() for transactional tables.
storage/myisam/myisamdef.h:
  Change my_once_flag to uint, as it stored different values than just 0/1
  Added 'open_flag' to store state given to mi_open()
storage/myisammrg/ha_myisammrg.cc:
  Add THR_LOCK_MERGE_PRIV to THR_LOCK_DATA to get MERGE locks sorted after other types of locks.
storage/myisammrg/myrg_locking.c:
  Remove windows specific code.
storage/myisammrg/myrg_open.c:
  Use HA_OPEN_MERGE_TABLE to mi_open().
  Set HA_OPEN_MERGE_TABLE for linked MyISAM tables.
storage/xtradb/buf/buf0buf.c:
  Fixed compiler warning
storage/xtradb/buf/buf0lru.c:
  Initialize variable that could be used not initialized.

Also fix missing zero termination in DBUG_PRINT, causing garbage output
in --debug output.

The main.ps_ddl test does SELECT * FROM mysql.general_log; that can be really
expensive with --valgrind if previous test cases put lots of data in the
general log since last server restart. Fix by truncating the log at test start.

BUG#26447 prefer a clustered key for an index scan, as secondary index is always slower
  ... which was fixed to cause
BUG#35850 queries take 50% longer
  ... and was reverted

and

BUG#39653 prefer a secondary index for an index scan, as clustered key is always slower
  ... which was fixed to cause
BUG#55656 mysqldump takes six days instead of half an hour
  ... and was amended with a special case workaround


sql/opt_range.cc:
  move get_index_only_read_time() into the handler class
sql/sql_select.cc:
  use cost not an index length when choosing a cheaper index

and mistakingly pulled in back for maria-5.1.50.

Fixes for bugs found by running test case for LP#608369 "Page: 1 Found wrong page type 0' on CHECK TABLE EXTENDED"
Fixed overflow when using long --debug=xxxxxx line.
Fixed that "mysqld --disable-debug --debug" works.
Ensure that MariaDB doesn't start if the Aria engine didn't start and we are using Aria for temporary tables.
More DBUG_ASSERT() and more info in debug log.


dbug/dbug.c:
  Fixed crash in mysqld caused by an overflow when using long --debug=xxxxxx line
sql/mysqld.cc:
  Fixed that "mysqld --disable-debug --debug" works.
  Documented myisam-recover=OFF option
storage/maria/ha_maria.cc:
  Ensure that MariaDB doesn't start if the Aria engine didn't start and we are using Aria for temporary tables.
storage/maria/ma_delete.c:
  Added missing write of changed key on node page.
  This could fix LP#608369 "Page: 1 Found wrong page type 0' on CHECK TABLE EXTENDED"
  Changed so that we log page numbers (not positions)
  Added KEY_OP_DEBUG_2 log entry to get more debug information into the log
storage/maria/ma_key_recover.c:
  Changed so that we log page numbers (not positions)
  In case of wrong page information after index_redo, dump pages to debug log
storage/maria/ma_loghandler.h:
  Added KEY_OP_DEBUG_2
storage/maria/ma_search.c:
  Changed so that we log page numbers (not positions)
storage/maria/ma_write.c:
  Changed so that we log page numbers (not positions)

sql/sql_parse.cc:
  Make some not commonly used functions with big local variables to separate functions to make default stack usage smaller.
  Decrease size of db_buff[] (Was bigger than needed)
  Allocate current_global_status_var with malloc().
storage/maria/ha_maria.cc:
  Don't allocate HA_CHECK on stack (it's > 100K)
storage/maria/ma_check.c:
  Removed duplicated code

- _ma_apply_redo_index: Assertion `page_offset != 0 && page_offset + length <= page_length' failed
Fixes one bug and one log assert when inserting rows:
- _ma_log_split: Assertion `org_length <= info->s->max_index_block_size' failed
- write_block_record:  Assertion '(data_length < MAX_TAIL_SIZE(block_size)' failed
Mark in recovery log where _ma_log_add() calls was done (for better debugging).

storage/maria/ma_bitmap.c:
  Don't write a head part on a tail page. (Caused an assert in write_block_record())
storage/maria/ma_delete.c:
  Mark in recovery log where _ma_log_add() calls was done
storage/maria/ma_key_recover.c:
  Mark in recovery log where _ma_log_add() calls was done
  Fixed not handled logging case for overfull index pages.
storage/maria/ma_key_recover.h:
  Mark in recovery log where _ma_log_add() calls was done
storage/maria/ma_loghandler.h:
  Mark in recovery log where _ma_log_add() calls was done
storage/maria/ma_rt_key.c:
  Mark in recovery log where _ma_log_add() calls was done
storage/maria/ma_write.c:
  Mark in recovery log where _ma_log_add() calls was done.
  Fixed wrong call to _ma_split_page() for overfull pages

The reason for this was that some bitmap test functions changed the bitmap, which caused problems when the same bitmap was used by multiple threads.


include/my_bitmap.h:
  Changed order of elements to get better alignment.
mysys/my_bitmap.c:
  Change bitmap test functions to not modify the bitmap.
  Fixed compiler errors in test_bitmap

Updated results for failing test cases (In all cases the estimated number of rows was different)

The condition over the outer tables now are extracted from
the on condition of any outer join. This condition is
saved in a special field of the JOIN_TAB structure for
the first inner table of the outer join. The condition
is checked before the first inner table is accessed. If 
it turns out to be false the table is not accessed at all
and a null complemented row is generated immediately.

the setting of optimizer switch for table elimination.