approved by: Marko
rb://681

Coalescing of free buf_page_t descriptors can prove to be one severe
bottleneck in performance of compression. One such workload where it
hurts badly is DROP TABLE. This patch removes buf_page_t allocations
from buf_buddy and uses ut_malloc instead.
In order to further reduce overhead of colaescing we no longer attempt
to coalesce a block if the corresponding free_list is less than 16 in
size.

This patch fixes a typo in a YaSSL error message.

TO POSITION FIRST CAN CAUSE DATA TO BE CORRUPTED".
 
ALTER TABLE MODIFY/CHANGE ... FIRST did nothing except renaming
columns if new version of the table had exactly the same 
structure as the old one (i.e. as result of such statement, names 
of columns changed their order as specified but data in columns 
didn't). The same thing happened for ALTER TABLE DROP COLUMN/ADD 
COLUMN statements which were supposed to produce new version of
table with exactly the same structure as the old version of table.
I.e. in the latter case the result was the same as if old column 
was renamed instead of being dropped and new column with default
as value being created.
 
Both these problems were caused by the fact that ALTER TABLE
implementation incorrectly interpreted both these situations as 
simple renaming of columns and assumed that in-place ALTER TABLE
algorithm could have been used for them.
 
This patch fixes this problem by ensuring that in cases when some
column is moved to the first position or some column is dropped
the default ALTER TABLE algorithm involving table copying is 
always used. This is achieved by detecting such situations in
mysql_prepare_alter_table() and setting Alter_info::change_level
to ALTER_TABLE_DATA_CHANGED for them.

mysql-test/r/alter_table.result:
  Added test for bug #12652385 - "61493: REORDERING COLUMNS TO
  POSITION FIRST CAN CAUSE DATA TO BE CORRUPTED".
mysql-test/t/alter_table.test:
  Added test for bug #12652385 - "61493: REORDERING COLUMNS TO
  POSITION FIRST CAN CAUSE DATA TO BE CORRUPTED".
sql/sql_table.cc:
  Changed mysql_prepare_alter_table() to detect situations in 
  which we some column moved to the first position or some column 
  is dropped and ensure that such ALTER TABLE statements won't
  be carried out using in-place algorithm. The latter could have 
  happened before this patch if new version of table had the same 
  structure as the old one (except the column names).

marko.makela@oracle.com-20110616072721-8bo92ctixq6eqavr

The buf_pool->zip_clean list is only needed for debugging, or for
recomputing buf_pool->page_hash when resizing the buffer pool. Buffer
pool resizing was never fully implemented. Remove the resizing code,
and define buf_pool->zip_clean only in debug builds.

buf_pool->zip_clean, buf_LRU_insert_zip_clean(): Enclose in
#if defined UNIV_DEBUG || UNIV_BUF_DEBUG.

buf_chunk_free(), buf_chunk_all_free(), buf_pool_shrink(),
buf_pool_page_hash_rebuild(), buf_pool_resize(): Remove (unreachable code).

rb:671 approved by Inaam Rana

There is an apparent problem with page_zip_clear_rec().
In btr_cur_optimistic_update() we do this:

	page_cur_delete_rec(page_cursor, index, offsets, mtr);
...
	rec = btr_cur_insert_if_possible(cursor, new_entry, 0/*n_ext*/, mtr);
	ut_a(rec); /* <- We calculated above the insert would fit */

The problem is that page_cur_delete_rec() could fill the modification
log while doing page_zip_clear_rec(), requiring recompression for the
btr_cur_insert_if_possible(). In a pathological case, the data could
fail to recompress.

page_zip_clear_rec(): Leave the page modification log alone. Only
clear the necessary fields.

rb:673 approved by Jimmy Yang

Replace UNIV_BLOB_NULL_DEBUG with UNIV_DEBUG||UNIV_BLOB_LIGHT_DEBUG. 
Fix known bogus failures.

btr_cur_optimistic_update(): If rec_offs_any_null_extern(), assert that
the current transaction is an incomplete transaction that is being
rolled back in crash recovery.

row_build(): If rec_offs_any_null_extern(), assert that the transaction
that last updated the record was recovered during crash recovery
(and will soon be rolled back).

btr_cur_compress_if_useful(), btr_compress(): Add the parameter ibool
adjust. If adjust=TRUE, adjust the cursor position after compressing
the page.

btr_lift_page_up(): Return a pointer to the father page.

BTR_KEEP_POS_FLAG: A new flag for btr_cur_pessimistic_update().

btr_cur_pessimistic_update(): If *big_rec != NULL and flags &
BTR_KEEP_POS_FLAG, keep the cursor positioned on the updated record.
Also, do not release the index tree x-lock if *big_rec != NULL.

btr_cur_mtr_commit_and_start(): Commits and restarts a
mini-transaction so that it will retain an x-lock on index->lock and
the page of the cursor. This is invoked when
btr_cur_pessimistic_update() returns *big_rec != NULL.

In all callers of btr_cur_pessimistic_update() that do not pass
BTR_KEEP_POS_FLAG, assert that *big_rec == NULL.

btr_cur_compress(): Unused function [in the built-in MySQL 5.1], remove.

page_rec_get_nth(): Return the nth record on the page (an inverse
function of page_rec_get_n_recs_before()). Refactored from
page_get_middle_rec().

page_get_middle_rec(): Invoke page_rec_get_nth().

page_cur_insert_rec_zip_reorg(): Make use of the page directory
shortcuts in page_rec_get_nth() instead of scanning the whole list of
records.

row_ins_clust_index_entry_by_modify(): Pass BTR_KEEP_POS_FLAG to
btr_cur_pessimistic_update().

row_ins_index_entry_low(): If row_ins_clust_index_entry_by_modify()
returns a big_rec, invoke btr_cur_mtr_commit_and_start() in order to
commit and start the mini-transaction without releasing the x-locks on
index->lock and the cursor page, and write the big_rec. Releasing the
page latch in mtr_commit() caused a race condition.

row_upd_clust_rec(): Pass BTR_KEEP_POS_FLAG to
btr_cur_pessimistic_update(). If it returns a big_rec, invoke
btr_cur_mtr_commit_and_start() in order to commit and start the
mini-transaction without releasing the x-locks on index->lock and the
cursor page, and write the big_rec. Releasing the page latch in
mtr_commit() caused a race condition.

sync_thread_add_level(): Add the parameter ibool relock. When TRUE,
bypass the latching order rules.

rw_lock_add_debug_info(): For nested X-lock requests, pass relock=TRUE
to sync_thread_add_level().

rb:678 approved by Jimmy Yang

Some ut_a(!rec_offs_any_null_extern()) assertion failures are indicating
genuine BLOB bugs, others are bogus failures when rolling back incomplete
transactions at crash recovery. This needs more work, and until I get a
chance to work on it, other testing must not be disrupted by this.

revno 2995.37.209
revision id marko.makela@oracle.com-20110518120508-qhn7vz814vn77v5k
parent marko.makela@oracle.com-20110517121555-lmple24qzxqkzep4
timestamp: Wed 2011-05-18 15:05:08 +0300
message:
  Fix a bogus UNIV_SYNC_DEBUG failure in the fix of Bug #59641
  or Oracle Bug #11766513.

  trx_undo_free_prepared(): Do not acquire or release trx->rseg->mutex.
  This code is invoked in the single-threaded part of shutdown, therefore
  a mutex is not needed.

SYNTAX TRIGGERS IN ANY WAY

Table with triggers which were using deprecated (5.0-only) syntax became
unavailable for any DML and DDL after upgrade to 5.1 version of server.
Attempt to execute any statement on such a table resulted in parsing
error reported. Since this included DROP TRIGGER and DROP TABLE
statements (actually, the latter was allowed but was not functioning
properly for such tables) it was impossible to fix the problem without
manual operations on .TRG and .TRN files in data directory.

The problem was that failure to parse trigger body (due to 5.0-only
syntax) when opening trigger file for a table prevented the table
from being open. This made all operations on the table impossible
(except DROP TABLE which due to peculiarity in its implementation
dropped the table but left trigger files around).

This patch solves this problem by silencing error which occurs when
we parse trigger body during table open. Error message is preserved
for the future use and table is marked as having a broken trigger.
We also try to analyze parse tree to recover trigger name, which
will be needed in order to drop the broken trigger. DML statements
which invoke triggers on the table marked as having broken trigger
are prohibited and emit saved error message. The same happens for
DDL which change triggers except DROP TRIGGER and DROP TABLE which
try their best to do what was requested. Table becomes no longer
marked as having broken trigger when last such trigger is dropped.

mysql-test/r/trigger-compat.result:
  Add results for test case for bug#45235
mysql-test/t/trigger-compat.test:
  Add test case for bug#45235.
sql/sp_head.cc:
  Added protection against MEM_ROOT double restoring to
  sp_head::restore_thd_mem_root() method. Since this
  method can be sometimes called twice during parsing
  of stored routine (the first time during normal flow
  of parsing, and the second time when a syntax error
  is detected) we need to shortcut execution of the
  method to avoid damaging MEM_ROOT by the second
  consecutive call to this method.
sql/sql_trigger.cc:
  Added error handler Deprecated_trigger_syntax_handler to 
  catch non-OOM errors during parsing of trigger body.
  
  Added handling of parse errors into method 
  Table_triggers_list::check_n_load().
sql/sql_trigger.h:
  Added new members to handle broken triggers and error messages.

row_build(): The record may contain null BLOB pointers when the server
is rolling back an insert that was interrupted by a server crash.

THE EVENT STATUS.

Any ALTER EVENT statement on a disabled event enabled it back
(unless this ALTER EVENT statement explicitly disabled the event).

The problem was that during processing of an ALTER EVENT statement
value of status field was overwritten unconditionally even if new
value was not specified explicitly. As a consequence this field
was set to default value for status which corresponds to ENABLE.

The solution is to check if status field was explicitly specified in
ALTER EVENT statement before assigning new value to status field.

mysql-test/r/events_bugs.result:
  test's result for Bug#11764334 was added.
mysql-test/t/events_bugs.test:
  new test for Bug#11764334 was added.
sql/event_db_repository.cc:
  mysql_event_fill_row() was modified: set value for status field
  in events tables only in case if statement CREATE EVENT
  is being processed or if this value was set in ALTER EVENT
  statement.
  Event_db_repository::create_event was modified: removed redundant
  setting of status field after return from call to mysql_event_fill_row().
sql/event_parse_data.h:
  Event_parse_data structure was modified: added flag
  status_changed that is set to true if status's value
  was changed in ALTER EVENT statement.
sql/sql_yacc.yy:
  Set flag status_changed if status was set in ALTER EVENT
  statement.

If UNIV_DEBUG or UNIV_BLOB_LIGHT_DEBUG is enabled, add
!rec_offs_any_null_extern() assertions, ensuring that records do not
contain null pointers to externally stored columns in inappropriate
places.

btr_cur_optimistic_update(): Assert !rec_offs_any_null_extern().
Incomplete records must never be updated or deleted. This assertion
will cover also the pessimistic route.

row_build(): Assert !rec_offs_any_null_extern(). Search tuples must
never be built from incomplete index entries.

row_rec_to_index_entry(): Assert !rec_offs_any_null_extern() unless
ROW_COPY_DATA is requested. ROW_COPY_DATA is used for
multi-versioning, and therefore it might be valid to copy the most
recent (uncommitted) version while it contains a null pointer to
off-page columns.

row_vers_build_for_consistent_read(),
row_vers_build_for_semi_consistent_read(): Assert !rec_offs_any_null_extern()
on all versions except the most recent one.

trx_undo_prev_version_build(): Assert !rec_offs_any_null_extern() on
the previous version.

rb:682 approved by Sunny Bains

Problem: in case of wrong data insert into indexed GEOMETRY fields 
(e.g. NULL value for a not NULL field) MyISAM reported 
"ERROR 126 (HY000): Incorrect key file for table; try to repair it"
due to misuse of the key deletion function.

Fix: always use R-tree key functions for R-tree based indexes
and B-tree key functions for B-tree based indexes.


mysql-test/r/gis-rtree.result:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - test result.
mysql-test/t/gis-rtree.test:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - test case.
storage/myisam/mi_update.c:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - handling update errors check for HA_ERR_NULL_IN_SPATIAL as well to be 
  consistent with mi_write();
    - always use keyinfo->ck_delete()/ck_insert() instead of _mi_ck_delete()/_mi_ck_write()
  to handle index properly, as it may be of B-tree or R-tree type.
storage/myisam/mi_write.c:
  Bug#11764487: myisam corruption with insert ignore and invalid spatial data
    - always use keyinfo->ck_delete() instead of _mi_ck_delete() to handle
  index properly, as it may be of B-tree or R-tree type.

  FAIL IN EMBEDDED SERVER

FreeBSD 64 bit needs the FP_X_DNML to fpsetmask() to prevent exceptions from
propagating into mysql (as a threaded application).
However fpsetmask() itself is deprecated in favor of fedisableexcept().
1. Fixed the #ifdef to check for FP_X_DNML instead of i386.
2. Added a configure.in check for fedisableexcept() and, if present,
   this function is called insted of the fpsetmask().
No need for new tests, as the existing tests cover this already.
Removed the affected tests from the experimental list.

The problem is that clients implementing the 4.0 version of the
protocol (that is, mysql-4.0) do not null terminate a string
at the end of the authentication packet. These clients denote
the end of the string with the end of the packet.

Although this goes against the documented (see MySQL Internals
ClientServer Protocol wiki) description of the protocol, these
old clients still need to be supported.

The solution is to support the documented and actual behavior
of the clients. If a client is using the pre-4.1 version of
the protocol, the end of a string in the authentication packet
can either be denoted with a null character or by the end of
the packet. This restores backwards compatibility with old
clients implementing either the documented or actual behavior.

sql/password.c:
  The scrambled message, as provided by the user, might not be
  properly null terminated. If this is the case, uninitialized
  memory past the end of the buffer could theoretically be
  accessed. To ensure that this is never the case, copy the
  scrambled message over to a null terminated auxiliar buffer.
sql/sql_connect.cc:
  Use different execution paths to read strings depending on the
  protocol being used. If version 4.0 of the protocol is used,
  end of string can be denoted with a NUL character or by the
  end of the packet.
  
  If there are not enough bytes left after the current position
  of the buffer to satisfy the current string, the string is
  considered to be empty. This is required because old clients
  do not send the password string field if the password is empty.

The problem is that although AIX implements bzero, its prototype
is not declared by default. Since AC_CHECK_FUNC(bzero) succeeds
even though a prototype is not declared, this breaks compilation
in C++ files where a prototype is required.

The solution is to only use bzero if a prototype is also declared.

configure.in:
  Check if bzero is declared. No need to specify the includes,
  unisted.h and strings.h are already part of AC_INCLUDES_DEFAULT.

will create multiple running events.

A CREATE IF NOT EXIST on an event that existed and was enabled caused
multiple instances of the event to run. Disabling the event didn't  help.
If the event was  dropped, the event stopped running, but when created
again, multiple instances of the event were still running. The only way
to get out of this situation was  to restart the server.

The problem was that Event_db_repository::create_event() didn't return
enough information to discriminate between situation when event didn't
exist and was created and when event did exist and was not created
(but a warning was emitted). As result in the latter case event
was added to in-memory queue of events second time. And this led to
unwarranted multiple executions of the same event.

The solution is to add out-parameter to Event_db_repository::create_event()
method which will signal that event was not created because it already
exists and so it should not be added to the in-memory queue.


mysql-test/r/events_bugs.result:
  Added results for test for Bug#12546938.
mysql-test/t/events_bugs.test:
  Added test for Bug#12546938.
sql/event_db_repository.cc:
  Event_db_repository::create_event was modified: set newly added out-parameter
  event_already_exists to true value if event wasn't created because event
  already existed and IF NOT EXIST clause was present.
sql/event_db_repository.h:
  Added out-parameter 'event_already_exists' to create_event() method.
sql/events.cc:
  Events::create_event was modified: insert new element into
  event queue only if event was actually created.

HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".

Attempt to update an InnoDB temporary table under LOCK TABLES
led to assertion failure in both debug and production builds
if this temporary table was explicitly locked for READ. The 
same scenario works fine for MyISAM temporary tables.

The assertion failure was caused by discrepancy between lock 
that was requested on the rows of temporary table at LOCK TABLES
time and by update operation. Since SQL-layer requested a 
read-lock at LOCK TABLES time InnoDB engine assumed that upcoming
statements which are going to be executed under LOCK TABLES will 
only read table and therefore should acquire only S-lock.
An update operation broken this assumption by requesting X-lock.

Possible approaches to fixing this problem are:

1) Skip locking of temporary tables as locking doesn't make any
   sense for connection-local objects.
2) Prohibit changing of temporary table locked by LOCK TABLES ... 
   READ.

Unfortunately both of these approaches have drawbacks which make 
them unviable for stable versions of server.

So this patch takes another approach and changes code in such way
that LOCK TABLES for a temporary table will always request write
lock. In 5.1 version of this patch switch from read lock to write
lock is done inside of InnoDBs handler methods as doing it on 
SQL-layer causes compatibility troubles with FLUSH TABLES WITH
READ LOCK.

mysql-test/suite/innodb/r/innodb_mysql.result:
  Added test for bug #11762012 - "54553: INNODB ASSERTS IN 
  HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".
mysql-test/suite/innodb/t/innodb_mysql.test:
  Added test for bug #11762012 - "54553: INNODB ASSERTS IN 
  HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".
mysql-test/suite/innodb_plugin/r/innodb_mysql.result:
  Added test for bug #11762012 - "54553: INNODB ASSERTS IN 
  HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".
mysql-test/suite/innodb_plugin/t/innodb_mysql.test:
  Added test for bug #11762012 - "54553: INNODB ASSERTS IN 
  HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".
storage/innobase/handler/ha_innodb.cc:
  Assume that a temporary table locked by LOCK TABLES can be updated
  even if it was only locked for read and therefore an X-lock should 
  be always requested for such tables.
storage/innodb_plugin/handler/ha_innodb.cc:
  Assume that a temporary table locked by LOCK TABLES can be updated
  even if it was only locked for read and therefore an X-lock should 
  be always requested for such tables.

Problem: MYSQL_BIN_LOG::reset_logs acquires mutexes in wrong order.
The correct order is first LOCK_thread_count and then LOCK_log. This function
does it the other way around. This leads to deadlock when run in parallel
with a thread that takes the two locks in correct order. For example, a thread
that disconnects will take the locks in the correct order.
Fix: change order of the locks in MYSQL_BIN_LOG::reset_logs:
first LOCK_thread_count and then LOCK_log.


mysql-test/suite/binlog/r/binlog_reset_master.result:
  added result file
mysql-test/suite/binlog/t/binlog_reset_master.test:
  Added test case that demonstrates deadlock because of wrong mutex order.
  The deadlock is between two threads:
   - RESET MASTER acquires mutexes in wrong order.
   - client thread shutdown code acquires mutexes in right order.
  Actually, this test case does not produce deadlock in 5.1, probably
  the client thread shutdown code does not hold both mutexes at the same
  time. However, the bug existed in 5.1 (mutexes are taken in the wrong
  order) so we push the test case to 5.1 too, to prevent future
  regressions.
sql/log.cc:
  Change mutex acquisition to the correct order:
  first LOCK_thread_count, then LOCK_log.
sql/mysqld.cc:
  Add debug code to synchronize test case.

Assertion happens due to missing NULL value check in
Item_func_round::fix_length_and_dec() function.
The fix: added NULL value check for second parameter.


mysql-test/r/func_math.result:
  test case
mysql-test/t/func_math.test:
  test case
sql/item_func.cc:
  added NULL value check for second parameter.

when selecting from I_S and views exist, in SP.

Symptoms: re-execution of prepared statement (or statement in a stored
routine) which read from one of I_S tables and which in order to fill
this I_S table had to open a view led to increasing memory consumption.

What happened in this situation was that during the process of view
opening for purpose of I_S filling view-related structures (like its
LEX) were allocated on persistent MEM_ROOT of prepared statement (or 
stored routine). Since this MEM_ROOT is not freed until prepared
statement deallocation (or expulsion of stored routine from the cache)
and code responsible for filling I_S is not able to re-use results of
view opening from previous executions this allocation ended up in
memory hogging.

This patch solves the problem by ensuring that when a view opened
for the purpose of I_S filling all its structures are allocated on
non-persistent runtime MEM_ROOT. This is achieved by activating a
temporary Query_arena bound to this MEM_ROOT.
Since this step makes impossible linking of view structures into
LEX of our prepared statement (or stored routine statement) this
patch also changes code filling I_S table to install a proxy LEX
before trying to open a view or a table. Consequently some code
which was responsible for backing-up/restoring parts of LEX when
view/table was opened during filling of I_S table became redundant
and was removed.

This patch doesn't contain test case for this bug as it is hard
to test memory hogging in our test suite.

Automerge mysql-5.1-gca into latest mysql-5.1.

              VM-WIN2003-32-A, SLES10-IA64-A 
      
The test case waits for master_pos_wait not to timeout, which
means that the deadlock between SQL and IO threads was 
succesfully and automatically dealt with.
      
However, very rarely, master_pos_wait reports a timeout. This
happens because the time set for master_pos_wait to wait was
too small (6 seconds). On slow test env this could be a 
problem.
      
We fix this by setting the timeout inline with the one used
in sync_slave_with_master (300 seconds). In addition we 
refactored the test case and refined some comments.

Issue:
While running embedded server, if client issues TEE command (\T foo/bar) and
"foo/bar" directory doesn't exist, it is suppose to give error. But it was
aborting.  This was happening because wrong error handler was being called.

Solution:
Modified calls to correct error handler. In embedded server case, there are 
two error handler (client and server) which are supposed to be called based 
on which context code is in. If it is in client context, client error handler
should be called otherwise server.

Test case:
Test case automation is not possible as current (following) code doesn't 
allow '\T' to be executed from command line (OR command read from a file):
[client/mysql.cc]
...
 static int
 com_tee(String *buffer __attribute__((unused)),
         char *line __attribute__((unused)))
 {
   char file_name[FN_REFLEN], *end, *param;

   if (status.batch) << THIS IS TRUE WHILE EXECUTING FROM COMMAND LINE.
      return 0;
 ...
So, not adding test case in GA. WIll add a test case in mysql-trunk after 
removing above code so that this could be properly tested before GA.

libmysqld/lib_sql.cc:
  Added code to call client/server error handler based on in control is in 
  client/server code respectively.
sql/mysql_priv.h:
  Added comments for THR_THD, THR_MALLOC keys.
sql/sql_class.cc:
  Function definition of new function restore_global to removes thread specific 
  data from stack (which was stored in store global).
sql/sql_class.h:
  Function declaration of new function restore_global.

               LEAK WITH PARTITIONED ARCHIVE TABLES

CHECK TABLE against archive table, when file descriptors
are exhausted, caused server crash.

Archive didn't handle errors when opening data file for
CHECK TABLE.

mysql-test/r/archive_debug.result:
  A test case for BUG#12402794.
mysql-test/t/archive_debug.test:
  A test case for BUG#12402794.
storage/archive/azio.c:
  A test case for BUG#12402794.
storage/archive/ha_archive.cc:
  Handle init_archive_reader() failure.