The progress reporting of InnoDB crash recovery was rather intermittent.
Nothing was reported during the single-threaded log record parsing, which
could consume minutes when parsing a large log. During log application,
there only was progress reporting in background threads that would be
invoked on data page read completion.

The progress reporting here will be detailed like this:

InnoDB: Starting crash recovery from checkpoint LSN=503549688
InnoDB: Parsed redo log up to LSN=1990840177; to recover: 124806 pages
InnoDB: Parsed redo log up to LSN=2729777071; to recover: 186123 pages
InnoDB: Parsed redo log up to LSN=3488599173; to recover: 248397 pages
InnoDB: Parsed redo log up to LSN=4177856618; to recover: 306469 pages
InnoDB: Multi-batch recovery needed at LSN 4189599815
InnoDB: End of log at LSN=4483551634
InnoDB: To recover: LSN 4189599815/4483551634; 307490 pages
InnoDB: To recover: LSN 4189599815/4483551634; 197159 pages
InnoDB: To recover: LSN 4189599815/4483551634; 67623 pages
InnoDB: Parsed redo log up to LSN=4353924218; to recover: 102083 pages
...
InnoDB: log sequence number 4483551634 ...

The previous messages "Starting a batch to recover" or
"Starting a final batch to recover" will be replaced by
"To recover: ... pages" messages.

If a batch lasts longer than 15 seconds, then there will be
progress reports every 15 seconds, showing the number of remaining pages.
For the non-final batch, the "To recover:" message includes two end LSN:
that of the batch, and of the recovered log. This is the primary measure
of progress. The batch will end once the number of pages to recover
reaches 0.

If recovery is possible in a single batch, the output will look like this,
with a shorter "To recover:" message that counts only the remaining pages:

InnoDB: Starting crash recovery from checkpoint LSN=503549688
InnoDB: Parsed redo log up to LSN=1998701027; to recover: 125560 pages
InnoDB: Parsed redo log up to LSN=2734136874; to recover: 186446 pages
InnoDB: Parsed redo log up to LSN=3499505504; to recover: 249378 pages
InnoDB: Parsed redo log up to LSN=4183247844; to recover: 306964 pages
InnoDB: End of log at LSN=4483551634
...
InnoDB: To recover: 331797 pages
...
InnoDB: log sequence number 4483551634 ...

We will also speed up recovery by improving the memory management and
implementing multi-threaded recovery of data pages that will not need
to be read into the buffer pool ("fake read"). Log application in the
"fake read" threads will be protected by an atomic being_recovered field
and exclusive buf_page_t::latch.

Recovery will reserve for data pages two thirds of the buffer pool,
or 256 pages, whichever is smaller. Previously, we could only use at most
one third of the buffer pool for buffered log records. This would typically
mean that with large buffer pools, recovery unnecessary consisted of
multiple batches.

If recovery runs out of memory, it will "roll back" or "rewind" the current
mini-transaction. The recv_sys.lsn and recv_sys.pages will correspond
to the "out of memory LSN", at the end of the previous complete
mini-transaction.

If recovery runs out of memory while executing the final recovery batch,
we can simply invoke recv_sys.apply(false) to make room, and resume
parsing.

If recovery runs out of memory before the final batch, we will scan
the redo log to the end (recv_sys.scanned_lsn) and check for any missing
or inconsistent files. If recv_init_crash_recovery_spaces() does not
report any potentially missing tablespaces, we can make use of the
already stored recv_sys.pages and only rewind to the "out of memory LSN".
Else, we must keep parsing and invoking recv_validate_tablespace()
until an error has been found or everything has been resolved, and
ultimatily rewind to to the checkpoint LSN.

recv_sys_t::pages_it: A cached iterator to recv_sys.pages

recv_sys_t::parse_mtr(): Remove an ATTRIBUTE_NOINLINE that would
prevent tail call optimization in recv_sys_t::parse_pmem().

recv_sys_t::parse(), recv_sys_t::parse_mtr(), recv_sys_t::parse_pmem():
Add template<bool store> parameter. Redo log record parsing
(store=false) is better specialized from store=true
(with bool if_exists) so that we can avoid some conditional branches
in frequently invoked low-level code.

recv_sys_t::is_memory_exhausted(): Remove. The special parse() status
GOT_OOM will report out-of-memory situation at the low level.

recv_sys_t::rewind(), page_recv_t::recs_t::rewind():
Remove all log starting with a specific LSN.

recv_scan_log(): Separate some code for only parsing, not storing log.
In rewound_lsn, remember the LSN at which last_phase=false recovery
ran out of memory. This is where the next call to recv_scan_log()
will resume storing the log. This replaces recv_sys.last_stored_lsn.

recv_sys_t::parse(): Evaluate the template parameter store in a few more
cases, to allow dead code to be eliminated at compile time.

recv_sys_t::scanned_lsn: The end of the log found by recv_scan_log().
The special value 1 means that recv_sys has been initialized but
no log has been parsed.

IORequest::write_complete(), IORequest::read_complete():
Replaces fil_aio_callback().

read_io_callback(), write_io_callback(): Replaces io_callback().

IORequest::fake_read_complete(), fake_io_callback(), os_fake_read():
Process a "fake read" request for concurrent recovery.

recv_sys_t::apply_batch(): Choose a number of successive pages
for a recovery batch.

recv_sys_t::erase(recv_sys_t::map::iterator): Remove log records for a
page whose recovery is not in progress. Log application threads
will not invoke this; they will only set being_recovered=-1 to indicate
that the entry is no longer needed.

recv_sys_t::garbage_collect(): Remove all being_recovered=-1 entries.

recv_sys_t::wait_for_pool(): Wait for some space to become available
in the buffer pool.

mlog_init_t::mark_ibuf_exist(): Avoid calls to
recv_sys::recover_low() via ibuf_page_exists() and buf_page_get_low().
Such calls would lead to double locking of recv_sys.mutex, which
depending on implementation could cause a deadlock. We will use
lower-level calls to look up index pages.

buf_LRU_block_remove_hashed(): Disable consistency checks for freed
ROW_FORMAT=COMPRESSED pages. Their contents could be uninitialized garbage.
This fixes an occasional failure of the test
innodb.innodb_bulk_create_index_debug.

Tested by: Matthias Leich

trx_purge_free_segment(), trx_purge_truncate_rseg_history():
Replace some unreachable code with debug assertions.
A buffer-fix does prevent pages from being evicted
from the buffer pool; see buf_page_t::can_relocate().

Tested by: Matthias Leich

trx_purge_truncate_history(): Only call trx_purge_truncate_rseg_history()
if the rollback segment is safe to process. This will avoid leaking undo
log pages that are not yet ready to be processed. This fixes a regression
that was introduced in
commit 0de3be8c (MDEV-30671).

trx_sys_t::any_active_transactions(): Separately count XA PREPARE
transactions.

srv_purge_should_exit(): Terminate slow shutdown if the history size
does not change and XA PREPARE transactions exist in the system.
This will avoid a hang of the test innodb.recovery_shutdown.

Tested by: Matthias Leich

The CODEOWNERS was added almost 3 years ago but never saw any adoption.
Only one person used it (me) to mark what files I maintain and for which
I wish to review commits. No other maintainers or code paths were added,
so clean it away for clarity.

collations

Analysis:
When we have negative index, the value in array_counter[] array is going to
be -1 at some point ( because in case of negative index in json path, the
initial value for a path with negative index is -<size_of_array>, and as we
move forward in array while parsing it and finding path, this value
increments). Since SKIPPED_STEP_MARK, is maximum uint value, it gets
compared to some int value in the array and eventually equates to -1
and messes with path.
Fix:
Make SKIPPED_STEP_MARK maximum of INT32.

Add Modern Perl headers. Perl 5.16 is still fairly
old from 2012.

Enable UTF-8, warnings and make script 'strict'

Small fixes for perlcritic reported problems and some crashes

I/O layer ":utf8" used at line 268, column 16.  Use ":encoding(UTF-8)" to get strict validation.  (Severity: 5)
"return" statement with explicit "undef" at line 806, column 4.  See page 199 of PBP.  (Severity: 5)
"return" statement with explicit "undef" at line 6844, column 4.  See page 199 of PBP.  (Severity: 5)
"return" statement with explicit "undef" at line 7524, column 4.  See page 199 of PBP.  (Severity: 5)
"return" statement with explicit "undef" at line 7527, column 4.  See page 199 of PBP.  (Severity: 5)
"return" statement with explicit "undef" at line 7599, column 4.  See page 199 of PBP.  (Severity: 5)
"return" statement with explicit "undef" at line 7602, column 4.  See page 199 of PBP.  (Severity: 5)
Expression form of "eval" at line 7784, column 4.  See page 161 of PBP.  (Severity: 5)
Expression form of "eval" at line 7806, column 4.  See page 161 of PBP.  (Severity: 5)
Glob written as <...> at line 8016, column 25.  See page 167 of PBP.  (Severity: 5)
"return" statement followed by "sort" at line 9195, column 60.  Behavior is undefined if called in scalar context.  (Severity: 5)
Expression form of "eval" at line 9846, column 10.  See page 161 of PBP.  (Severity: 5)

buf_read_page_low(): Remove an error message and a debug assertion
that can be triggered when using innodb_page_size=4k and
innodb_file_per_table=0. In that case, buf_read_ahead_linear()
may be invoked on page 255, which is one less than the first
page of the doublewrite buffer (256).

fil_space_t::flush_freed(): Renamed from buf_flush_freed_pages();
this is a backport of aa458506 from 10.6.
Invoke log_write_up_to() on last_freed_lsn, instead of avoiding
the operation when the log has not yet been written.
A more costly alternative would be that log_checkpoint() would invoke
this function on every affected tablespace.

buf_read_ahead_linear(): Correct some calculations that were broken
in commit b1ab211d (MDEV-15053).

Thanks to Daniel Black for providing a test case and initial debugging.

Tested by: Matthias Leich

btr_cur_need_opposite_intention(): Check also page_zip_available()
so that we will escalate to exclusive index latch when a non-leaf
page may have to be split further due to ROW_FORMAT=COMPRESSED page
overflow.

Tested by: Matthias Leich

best_access_path(): Simplify the logic for computing fanout.
This fixes up commit 4329ec5d

Command lsof can fail on Debian install.
Revert logic more like old one to make sure that there is no failing
and still does don't boundce on shellcheck.

Debian install scripts if-clauses are not formatted
as they should be. This commit formats Debian
Pre and Post script if-clauses.

Commit fixes several ShellCheck found problems in Debian
Pre- and Postscripts.

Debian script mariadb-server-10.6.postrm contains shellcheck
Fixed problems are:
  https://www.shellcheck.net/wiki/SC2166 -- Prefer [ p ] && [ q ] as [ p -a q...
  https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: /usr/share/debconf...

Debian script mariadb-server-10.6.postinst contains shellcheck
Fixed problems are:
  https://www.shellcheck.net/wiki/SC2129 -- Consider using { cmd1; cmd2; } >>...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: /usr/share/debconf...
  https://www.shellcheck.net/wiki/SC1072 -- Expected test to end here (don't ...
  https://www.shellcheck.net/wiki/SC1073 -- Couldn't parse this test expressi...
  https://www.shellcheck.net/wiki/SC1009 -- The mentioned syntax error was in...

Debian script mariadb-server-10.6.preinst contains shellcheck
Fixed problems are:
  https://www.shellcheck.net/wiki/SC2231 -- Quote expansions in this for loop...
  https://www.shellcheck.net/wiki/SC2166 -- Prefer [ p ] && [ q ] as [ p -a q...
  https://www.shellcheck.net/wiki/SC2001 -- See if you can use ${variable//se...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: /usr/share/debconf...
  https://www.shellcheck.net/wiki/SC1007 -- Remove space after = if trying to...

There is several misindentation inside Debian post and pre
installation scripts. False indentation with space
as indent space should be 2 and indentation with tabs.

Derived table creation code would call Field::make_new_field() which would
memcpy the Field object from the source table, including Field::read_stats.

But the temp. table as a whole had table->stats_is_read=false. Which was
correct but not consistent with Field::read_stats and caused an assertion.

Fixed by making sure that Field::read_stats=NULL for fields in the new
temporary (i.e. work) tables.

The problem, introduced in patch for MDEV-26301:

When check_join_cache_usage() decides not to use join buffer, it must
adjust the access method accordingly. For BNL-H joins this means switching
from pseudo-"ref access"(with index=MAX_KEY) to some other access method.

Failing to do this will cause assertions down the line when code that is
not aware of BNL-H will try to initialize index use for ref access with
index=MAX_KEY.

The fix is to follow the regular code path to disable the join buffer for
the join_tab ("goto no_join_cache") instead of just returning from
check_join_cache_usage().