bug#41023 maria: Server asserts in sysbench OLTP_RO test 
bug#40888 maria: Server crashes in sysbench OLTP_RW test at lf_alloc-pin.c:513
bug#40892 maria: Livelock in sysbench OLTP_RW test 
bug#40895 maria: Server crashes in sysbench OLTP_RO test at lf_alloc-pin.c:367
bug#40890 maria: Server crashes in sysbench OLTP_RW test at ctype-bin.c:8

Yet another strict aliasing issue

we cannot access a trn after we put it in the pool

mysql-test/suite/maria/t/maria_showlog_error.test:
  test which can't run in embedded mode
storage/innobase/handler/ha_innodb.cc:
  compiler warning

Fixing Valgrind error: deadlock detector was set to random; seen in many InnoDB tests like innodb-replace.test

storage/innobase/handler/ha_innodb.cc:
  Fixing Valgrind error: deadlock detector was set to random. Making the recently added lines closer
  to the InnoDB style (tabs not spaces)

mysys/my_init.c:
  Fixed link error when compiling without thread support
sql/item_create.cc:
  Fixed compiler warning
sql/mysqld.cc:
  Fixed compile error on windows
sql/protocol.cc:
  Fixed compiler warning
sql/sql_class.cc:
  Fixed compiler warning
sql/sql_class.h:
  Fixed compiler warning
storage/myisam/mi_open.c:
  Fixed compiler warning
storage/myisammrg/ha_myisammrg.cc:
  Fixed compiler warning (shadow variable)

This writes a warning on stderr if one uses mutex in different order,
like if one in one case would lock mutex in the order A,B and in another case
would lock mutex in the order B,A
This is inspired by and loosely based on the LOCKDEP patch by Jonas
Wrong mutex order is either fixed or mutex are marked with MYF_NO_DEADLOCK_DETECTION
if used inconsistently (need to be fixed by server team)

KNOWN_BUGS.txt:
  Added information that one need to dump and restore Maria tables
include/hash.h:
  Added prototype function for walking over all elements in a hash
include/my_pthread.h:
  Added my_pthread_mutex_init() and my_pthread_mutex_lock(); These should be used if one wants to disable mutex order checking.
  Changed names of the nonposix mutex_init functions to not conflict with my_phread_mutex_init()
  Added and extended structures for mutex deadlock detection.
  New arguments to sage_mutex_init() and safe_mutex_lock() to allow one to disable mutex order checking.
  Added variable 'safe_mutex_deadlock_detector' to enable/disable deadlock detection for all pthread_mutex_init()
mysys/Makefile.am:
  Added cleaning of test files
  Added test_thr_mutex
mysys/hash.c:
  Added hash_iterate() to iterate over all elements in a hash
  More comments
mysys/my_init.c:
  Added calls to destory all mutex uses by mysys()
  Added waiting for threads to end before calling TERMINATE() to list not freed memory
mysys/my_pthread.c:
  Changed names to free my_pthread_mutex_init() for mutex-lock-order-checking
mysys/my_sleep.c:
  Fixed too long wait if using 1000000L as argument
mysys/my_thr_init.c:
  Mark THR_LOCK_threads and THR_LOCK_malloc to not have mutex deadlock detection.
  (We can't have it enabled for this as these are internal mutex used by the detector  
  Call my_thread_init() early as we need thread specific variables enabled for the following pthread_mutex_init()
  Move code to wait for threads to end to my_wait_for_other_threads_to_die()
  Don't destroy mutex and conditions unless all threads have died
  Added my_thread_destroy_mutex() to destroy all mutex used by the mysys thread system
  Name the thread specific mutex as "mysys_var->mutex"
  Added my_thread_var_mutex_in_use() to return pointer to mutex in use or 0 if thread variables are not initialized
mysys/mysys_priv.h:
  Added prototypes for functions used internally with mutex-wrong-usage detection
mysys/thr_mutex.c:
  Added runtime detection of mutex used in conflicting order
  See WL#3262 or test_thr_mutex.c for examples
  The base idea is for each mutex have two hashes:
  - mutex->locked_mutex points to all mutex used after this one
  - mutex->used_mutex points to all mutex which has this mutex in it's mutex->locked_mutex
  There is a wrong mutex order if any mutex currently locked before this mutex is in the mutex->locked_mutex hash
sql/event_queue.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/event_scheduler.cc:
  Declare the right order to take the mutex
sql/events.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/ha_ndbcluster_binlog.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/log.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/mysqld.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Added call to free sp cache.
  Add destruction of LOCK_server_started and COND_server_started.
  Added register_mutex_order() function to register in which order mutex should be taken
  (to initiailize mutex_deadlock_detector).
  Added option to turn off safe_mutex_deadlock_detector
sql/protocol.cc:
  Fixed wrong argument to DBUG_PRINT (found by valgrind)
sql/rpl_mi.cc:
  Mark mutex used inconsistently (need to be fixed by server team)
sql/set_var.cc:
  Remove wrong locking of LOCK_global_system_variables when reading and setting log variables
  (would cause inconsistent mutex order).
  Update global variables outside of logger.unlock() as LOCK_global_system_variables has to be taken before logger locks
  Reviewed by gluh
sql/sp_cache.cc:
  Added function to destroy mutex used by sp cache
sql/sp_cache.h:
  Added function to destroy mutex used by sp cache
sql/sql_class.cc:
  Use pthread_mutex_trylock instead of pthread_mutex_unlock() when sending kill signal to thread
  This is needed to avoid wrong mutex order as normally one takes 'current_mutex' before mysys_var->mutex.
  Register order in which LOCK_delete and mysys_var->mutex is taken
sql/sql_insert.cc:
  Give a name for Delayed_insert::mutex
  Mark mutex used inconsistently (need to be fixed by server team)
  Move closing of tables outside of di->mutex (to avoid wrong mutex order)
sql/sql_show.cc:
  Don't keep LOCK_global_system_variables locked over value->show_type() as this leads to wrong mutex order
storage/innobase/handler/ha_innodb.cc:
  Disable safe_muted_deadlock_detector for innobase intern mutex (to speed up page cache initialization)
storage/maria/ha_maria.cc:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
  This is needed to avoid locking mutex in wrong order
storage/maria/ha_maria.h:
  Added flag to ha_maria::info() to signal if we need to lock table share or not.
storage/maria/ma_close.c:
  Destroy key_del_lock
  Simplify freeing ftparser_param
storage/maria/ma_key.c:
  Better comment
storage/maria/ma_loghandler.c:
  Mark mutex used inconsistently (need to be fixed by sanja)
storage/maria/ma_state.c:
  More comments
storage/maria/ma_test1.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)
storage/maria/ma_test2.c:
  Ensure that safe_mutex_deadlock_detector is always on (should be, this is just for safety)

Ensure that if we send 'sigint' to mysqld started with --gdb it's taken down nicely
Clear THR_THD when thread ends to get a valid stack trace if we die late
Bug #37869 mysqld stopped reacting to SIGINT when run with --gdb

mysqlslap: fix a crash when mysql_store_result() fails

client/mysqlslap.c:
  fix a crash
dbug/dbug.c:
  only do safemalloc checks if a function is selected
mysql-test/mysql-test-run.pl:
  it's easier to add new gdb parameters this way
storage/maria/ma_open.c:
  typo in a comment

(BUG#41127: Maria: assertion when SHOW ENGINE MARIA LOGS and missing logs)

mysql-test/suite/maria/r/maria_showlog_error.result:
  test suite for the BUG#41127
mysql-test/suite/maria/t/maria_showlog_error.test:
  test suite for the BUG#41127
storage/maria/ha_maria.cc:
  Do not use MY_WME in the stat call which errors we process on high level.

an item was evaluated unnecessary, fix that by checking preconditions before evaluating the item

sql/sql_select.cc:
  an item was evaluated unnecessary, fix that by checking preconditions before evaluating the item

bug is actually a weirdness in test system, so moving test portion into maria_notembedded.test

mysql-test/suite/maria/r/maria.result:
  result update
mysql-test/suite/maria/r/maria_notembedded.result:
  result update
mysql-test/suite/maria/t/maria.test:
      Removing test portion into maria_notembedded.test. Explanation below.
      Running maria.test with --mem --embedded --valgrind I got:
      ==378== Invalid read of size 4
      ==378==    at 0x857C755: _lf_pinbox_real_free (lf_alloc-pin.c:342)
      ==378==    by 0x857C640: _lf_pinbox_free (lf_alloc-pin.c:267)
      ==378==    by 0x857D50C: ldelete (lf_hash.c:226)
      ==378==    by 0x857DBC1: lf_hash_delete (lf_hash.c:429)
      ==378==    by 0x8243D82: unlock_lock_and_free_resource
      (waiting_threads.c:676)
      ==378==    by 0x8244A79: wt_thd_release (waiting_threads.c:948)
      ==378==    by 0x83F301D: wt_thd_release_self (trnman.c:98)
      ==378==    by 0x83F3D49: trnman_end_trn (trnman.c:431)
      ==378==    by 0x837C437: ma_commit (ma_commit.c:60)
      ==378==    by 0x835F082: ha_maria::external_lock(THD*, int)
      (ha_maria.cc:2387)
      ==378==    by 0x84D1011: handler::ha_external_lock(THD*, int)
      (handler.cc:4433)
      ==378==    by 0x856DA92: unlock_external(THD*, st_table**, unsigned)
      (lock.cc:786)
      ==378==    by 0x856DD6F: mysql_unlock_tables(THD*, st_mysql_lock*)
      (lock.cc:389)
      ==378==    by 0x8274A57: close_thread_tables(THD*) (sql_base.cc:1307)
      ==378==    by 0x82ABEC3: unlock_locked_tables(THD*) (sql_parse.cc:99)
      ==378==    by 0x82B0BB3: mysql_execute_command(THD*)
      (sql_parse.cc:3372)
      ==378==  Address 0x5894a0c is 164 bytes inside a block of size 184
      free'd
      ==378==    at 0x40233FC: free (vg_replace_malloc.c:323)
      ==378==    by 0x823CF92: my_thread_end (my_thr_init.c:348)
      ==378==    by 0x8200FEE: mysql_thread_end (libmysql.c:250)
      ==378==    by 0x81A9487: send_one_query (mysqltest.c:535)
      ==378==    by 0x4050111: start_thread (in /lib/libpthread-2.5.so)
      ==378==    by 0x41A52ED: clone (in /lib/libc-2.5.so)
      
      (among other errors). The line where the invalid read happens is:
        if (available_stack_size(&pinbox, *pins->stack_ends_here) >
      alloca_size)
      and stack_ends_here was previously set here:
        el->stack_ends_here= & my_thread_var->stack_ends_here;
      The mysqltest "send" command, in embedded mode, is implemented this way
      (mysqltest.c:do_send_query()): a *new OS thread* is created in
      mysqltest, and does this:
      
        mysql_thread_init();
        VOID(mysql_send_query(&cn->mysql, cn->cur_query,
      cn->cur_query_len));
        mysql_thread_end();
      
      So, con_d "send insert t1 values (2)" creates a new OS thread, which
      gets a thread-specific data (mysql_thread_init()), then sends the
      query; in particular, this sets el->stack_ends_here to a pointer inside
      the thread-specific data (see my_thread_var above), that is, thd->pins
      depends on the thread-specific data.
      This sent "insert" blocks as expected, thread-specific data is free()d
      (my_thread_end() above), OS thread terminates.
      Then "default" connection runs "insert t1 values (3)"
      which blocks on the 3 already inserted by the con_d.
      This blocking (see waiting_threads.c) creates a WT_RESOURCE, which has,
      as owner, con_d's WT_THD.
      When con_d calls "unlock tables" (see stack trace), it wants to release
      this resource, which involves con_d's thd->pins, so it ends up reading
      stack_ends_here which points inside the freed thread-specific data =>
      Valgrind error, crashes...
      So this is an effect of having one single connection (con_d) for which
      two queries:
      send insert t1 values (2)
      unlock tables
      are done by different OS threads. That is indeed weird design of
      mysqltest, and it breaks on the dependency of lf_alloc-pin.c on
      thread-specific data. It is maybe already explained in those comments
      in lf_alloc-pin.c:
      "  It is assumed that pins belong to a THD and are not transferable
        between THD's (LF_PINS::stack_ends_here being a primary reason
        for this limitation)."
      "    It is assumed that pins belong to a thread and are not
      transferable
          between threads."
      
      In correct usage of libmysqld (no two queries sent for one connection
      by two threads), this problem would not happen, so I call it a
      deficiency of the test system.
mysql-test/suite/maria/t/maria_notembedded.test:
  moving test portion into maria_notembedded.test

Manually applying the patch for BUG40954 "Crash in MyISAM index code with concurrency test using partitioned tables"
so that it stops crashing pushbuild2/5.1-maria and we can see the other failures which it hid.

mysql-test/r/partition.result:
  Manually applying the patch for BUG40954 "Crash in MyISAM index code with concurrency test using partitioned tables"
mysql-test/t/partition.test:
  Manually applying the patch for BUG40954 "Crash in MyISAM index code with concurrency test using partitioned tables"
sql/ha_partition.cc:
  Manually applying the patch for BUG40954 "Crash in MyISAM index code with concurrency test using partitioned tables"

so cause timeouts. Probably normal as ALTER TABLE does many syncs in Maria, but have to mark them "big"
for pushbuild2.

mysql-test/suite/parts/t/partition_alter2_1_maria.test:
  test takes 15 minutes with Maria (ALTER TABLE does many syncs in Maria)
mysql-test/suite/parts/t/partition_alter2_2_maria.test:
  test takes 15 minutes with Maria (ALTER TABLE does many syncs in Maria)

(i.e. file's size % 8192 != 0). Don't assert on this, Recovery is supposed to fix it.

Under Windows one can get \ instead of / in file names in error messages (fix for pushbuild warning)

mysql-test/lib/mtr_report.pl:
  Under Windows one can get \ instead of / in file names in error messages

      Scenario of the BUG#40731 ("Maria: hang (probably in page cache) under concurrency"):
      T1: Disable logging for the table
      T1: Start inserting into the table
      T2: Tries to lock the table so waits.
      T2: Tries unlock and relock during the process see that the table has disabled logging and reenables it
      T1: Got DBUG_ASSERT because suddenly start use table with transaction switched on which is not expected during bulk insert

storage/maria/ma_pagecache.c:
  Page type print added for debugging purposes.
storage/maria/ma_recovery.c:
  Check that it was this thred which switched off logging (transactional mode).
storage/maria/maria_def.h:
  Flag for controling which thread switched off transactiona mode for the table added.

Merge of 5.1-main into 5.1-maria. There were no changes to storage/myisam, or mysql-test/t/*myisam*.
However there were three new tests mysql-test/suite/parts/t/partition*myisam.test, of which I make here
copies for Maria.

storage/maria/trnman.c:
  During Maria's checkpoint, we walk the list of active transactions; in this list we may find a transaction with a short_id of 0 which means "uninitialized" (is being created right now) and want to ignore this transaction. Such short_id is set under trn->state_lock, so use this mutex to reliably read short_id during checkpoint.

storage/maria/ma_pagecache.c:
  Added correct reaction on find operation status during writing the page.

storage/maria/ma_pagecache.c:
  Added correct reaction on find operation status during writing the page.

            
Innodb monitor could cause a server crash because of invalid access to a
shared variable in a concurrent environment.
                  
This patch adds a guard to protect against crashes but not against
inconsistent values because of performance reasons.

sql/sql_class.cc:
  * Attempt snapshot of static memory pointer proc_info to avoid null pointers.

where it was forgotten to remove them.

refining the regression test to avoid explicit innodb engine in create.