Skip to content

M
mariadb
Commit 73a98018 authored by arjen@co3064164-a.bitbike.com's avatar arjen@co3064164-a.bitbike.com
Browse files
Options

Improved improvement of Internals story and location.

parent 176631c7
Show whitespace changes
Inline Side-by-side
Showing with 767 additions and 764 deletions
Docs/manual.texi
View file @ 73a98018
......@@ -44261,1003 +44261,1006 @@ contains an Eiffel wrapper written by Michael Ravits.
@chapter Extending MySQL
@menu
* MySQL internals:: MySQL Internals
* Adding functions:: Adding New Functions to MySQL
* Adding procedures:: Adding New Procedures to MySQL
* MySQL internals:: MySQL Internals
@end menu
@node Adding functions, Adding procedures, Extending MySQL, Extending MySQL
@section Adding New Functions to MySQL
@cindex functions, new
@cindex adding, new functions
@cindex user-defined functions, adding
@cindex UDFs, defined
@cindex functions, user-defined
@node MySQL internals, Adding functions, Extending MySQL, Extending MySQL
@section MySQL Internals
There are two ways to add new functions to MySQL:
@cindex internals
@cindex threads
@itemize @bullet
@item You can add the function through the user-definable function (UDF)
interface. User-definable functions are added and removed dynamically using
the @code{CREATE FUNCTION} and @code{DROP FUNCTION} statements.
@xref{CREATE FUNCTION, , @code{CREATE FUNCTION}}.
This chapter describes a lot of things that you need to know when
working on the MySQL code. If you plan to contribute to MySQL
development, want to have access to the bleeding-edge in-between
versions code, or just want to keep track of development, follow the
instructions in @xref{Installing source tree}.
If you are interested in MySQL internals, you should also subscribe
to our @code{internals} mailing list. This list is relatively low
traffic. For details on how to subscribe, please see
@ref{Mailing-list}.
All developers at MySQL AB are on the @code{internals} list and we
help other people who are working on the MySQL code. Feel free to
use this list both to ask questions about the code and to send
patches that you would like to contribute to the MySQL project!
@item You can add the function as a native (built in) MySQL function.
Native functions are compiled into the @code{mysqld} server and become
available on a permanent basis.
@end itemize
@menu
* MySQL threads:: MySQL threads
* MySQL test suite:: MySQL test suite
@end menu
Each method has advantages and disadvantages:
@itemize @bullet
@item
If you write a user-definable function, you must install the object file
in addition to the server itself. If you compile your function into the
server, you don't need to do that.
@item
You can add UDFs to a binary MySQL distribution. Native functions
require you to modify a source distribution.
@item
If you upgrade your MySQL distribution, you can continue to use your
previously installed UDFs. For native functions, you must repeat your
modifications each time you upgrade.
@end itemize
@node MySQL threads, MySQL test suite, MySQL internals, MySQL internals
@subsection MySQL Threads
Whichever method you use to add new functions, they may be used just like
native functions such as @code{ABS()} or @code{SOUNDEX()}.
The MySQL server creates the following threads:
@menu
* CREATE FUNCTION:: @code{CREATE FUNCTION/DROP FUNCTION} Syntax
* Adding UDF:: Adding a new user-definable function
* Adding native function:: Adding a new native function
@end menu
@itemize @bullet
@item
The TCP/IP connection thread handles all connection requests and
creates a new dedicated thread to handle the authentication and
and SQL query processing for each connection.
@node CREATE FUNCTION, Adding UDF, Adding functions, Adding functions
@subsection @code{CREATE FUNCTION/DROP FUNCTION} Syntax
@item
On Windows NT there is a named pipe handler thread that does the same work as
the TCP/IP connection thread on named pipe connect requests.
@findex CREATE FUNCTION
@findex DROP FUNCTION
@findex UDF functions
@findex User-defined functions
@findex Functions, user-defined
@item
The signal thread handles all signals. This thread also normally handles
alarms and calls @code{process_alarm()} to force timeouts on connections
that have been idle too long.
@example
CREATE [AGGREGATE] FUNCTION function_name RETURNS @{STRING|REAL|INTEGER@}
SONAME shared_library_name
@item
If @code{mysqld} is compiled with @code{-DUSE_ALARM_THREAD}, a dedicated
thread that handles alarms is created. This is only used on some systems where
there are problems with @code{sigwait()} or if one wants to use the
@code{thr_alarm()} code in ones application without a dedicated signal
handling thread.
DROP FUNCTION function_name
@end example
@item
If one uses the @code{--flush_time=#} option, a dedicated thread is created
to flush all tables at the given interval.
A user-definable function (UDF) is a way to extend MySQL with a new
function that works like native (built in) MySQL functions such as
@code{ABS()} and @code{CONCAT()}.
@item
Every connection has its own thread.
@code{AGGREGATE} is a new option for MySQL Version 3.23. An
@code{AGGREGATE} function works exactly like a native MySQL
@code{GROUP} function like @code{SUM} or @code{COUNT()}.
@item
Every different table on which one uses @code{INSERT DELAYED} gets its
own thread.
@code{CREATE FUNCTION} saves the function's name, type, and shared library
name in the @code{mysql.func} system table. You must have the
@strong{insert} and @strong{delete} privileges for the @code{mysql} database
to create and drop functions.
@item
If you use @code{--master-host}, a slave replication thread will be
started to read and apply updates from the master.
@end itemize
All active functions are reloaded each time the server starts, unless
you start @code{mysqld} with the @code{--skip-grant-tables} option. In
this case, UDF initialisation is skipped and UDFs are unavailable.
(An active function is one that has been loaded with @code{CREATE FUNCTION}
and not removed with @code{DROP FUNCTION}.)
@code{mysqladmin processlist} only shows the connection, @code{INSERT DELAYED},
and replication threads.
For instructions on writing user-definable functions, see @ref{Adding
functions}. For the UDF mechanism to work, functions must be written in C or
C++, your operating system must support dynamic loading and you must have
compiled @code{mysqld} dynamically (not statically).
Note that to make @code{AGGREGATE} work, you must have a
@code{mysql.func} table that contains the column @code{type}. If this
is not the case, you should run the script
@code{mysql_fix_privilege_tables} to get this fixed.
@node MySQL test suite, , MySQL threads, MySQL internals
@subsection MySQL Test Suite
@cindex mysqltest, MySQL Test Suite
@cindex testing mysqld, mysqltest
@node Adding UDF, Adding native function, CREATE FUNCTION, Adding functions
@subsection Adding a New User-definable Function
Until recently, our main full-coverage test suite was based on proprietary
customer data and for that reason has not been publicly available. The only
publicly available part of our testing process consisted of the @code{crash-me}
test, a Perl DBI/DBD benchmark found in the @code{sql-bench} directory, and
miscellaneous tests located in @code{tests} directory. The lack of a
standardised publicly available test suite has made it difficult for our users,
as well developers, to do regression tests on the MySQL code. To
address this problem, we have created a new test system that is included in
the source and binary distributions starting in Version 3.23.29.
@cindex adding, user-definable functions
@cindex user-defined functions, adding
@cindex functions, user-definable, adding
The current set of test cases doesn't test everything in MySQL, but it
should catch most obvious bugs in the SQL processing code, OS/library
issues, and is quite thorough in testing replication. Our eventual goal
is to have the tests cover 100% of the code. We welcome contributions
to our test suite. You may especially want to contribute tests that
examine the functionality critical to your system, as this will ensure
that all future MySQL releases will work well with your
applications.
@menu
* UDF calling sequences:: UDF calling sequences
* UDF arguments:: Argument processing
* UDF return values:: Return values and error handling
* UDF compiling:: Compiling and installing user-definable functions
* running mysqltest:: Running the MySQL Test Suite
* extending mysqltest:: Extending the MySQL Test Suite
* Reporting mysqltest bugs:: Reporting Bugs in the MySQL Test Suite
@end menu
For the UDF mechanism to work, functions must be written in C or C++ and your
operating system must support dynamic loading. The MySQL source
distribution includes a file @file{sql/udf_example.cc} that defines 5 new
functions. Consult this file to see how UDF calling conventions work.
@node running mysqltest, extending mysqltest, MySQL test suite, MySQL test suite
@subsubsection Running the MySQL Test Suite
For @code{mysqld} to be able to use UDF functions, you should configure MySQL
with @code{--with-mysqld-ldflags=-rdynamic} The reason is that to on
many platforms (including Linux) you can load a dynamic library (with
@code{dlopen()}) from a static linked program, which you would get if
you are using @code{--with-mysqld-ldflags=-all-static} If you want to
use an UDF that needs to access symbols from @code{mysqld} (like the
@code{methaphone} example in @file{sql/udf_example.cc} that uses
@code{default_charset_info}), you must link the program with
@code{-rdynamic} (see @code{man dlopen}).
The test system consist of a test language interpreter
(@code{mysqltest}), a shell script to run all
tests(@code{mysql-test-run}), the actual test cases written in a special
test language, and their expected results. To run the test suite on
your system after a build, type @code{make test} or
@code{mysql-test/mysql-test-run} from the source root. If you have
installed a binary distribution, @code{cd} to the install root
(eg. @code{/usr/local/mysql}), and do @code{scripts/mysql-test-run}.
All tests should succeed. If not, you should try to find out why and
report the problem if this is a bug in MySQL.
@xref{Reporting mysqltest bugs}.
For each function that you want to use in SQL statements, you should define
corresponding C (or C++) functions. In the discussion below, the name
``xxx'' is used for an example function name. To distinquish between SQL and
C/C++ usage, @code{XXX()} (uppercase) indicates a SQL function call, and
@code{xxx()} (lowercase) indicates a C/C++ function call.
If you have a copy of @code{mysqld} running on the machine where you want to
run the test suite you do not have to stop it, as long as it is not using
ports @code{9306} and @code{9307}. If one of those ports is taken, you should
edit @code{mysql-test-run} and change the values of the master and/or slave
port to one that is available.
The C/C++ functions that you write to implement the interface for
@code{XXX()} are:
You can run one individual test case with
@code{mysql-test/mysql-test-run test_name}.
@table @asis
@item @code{xxx()} (required)
The main function. This is where the function result is computed.
The correspondence between the SQL type and return type of your C/C++
function is shown below:
If one test fails, you should test running @code{mysql-test-run} with
the @code{--force} option to check if any other tests fails.
@multitable @columnfractions .2 .8
@item @strong{SQL type} @tab @strong{C/C++ type}
@item @code{STRING} @tab @code{char *}
@item @code{INTEGER} @tab @code{long long}
@item @code{REAL} @tab @code{double}
@end multitable
@item @code{xxx_init()} (optional)
The initialisation function for @code{xxx()}. It can be used to:
@node extending mysqltest, Reporting mysqltest bugs, running mysqltest, MySQL test suite
@subsubsection Extending the MySQL Test Suite
You can use the @code{mysqltest} language to write your own test cases.
Unfortunately, we have not yet written full documentation for it - we plan to
do this shortly. You can, however, look at our current test cases and use
them as an example. The following points should help you get started:
@itemize @bullet
@item
Check the number of arguments to @code{XXX()}.
@item
Check that the arguments are of a required type or, alternatively,
tell MySQL to coerce arguments to the types you want when
the main function is called.
The tests are located in @code{mysql-test/t/*.test}
@item
Allocate any memory required by the main function.
A test case consists of @code{;} terminated statements and is similar to the
input of @code{mysql} command line client. A statement by default is a query
to be sent to MySQL server, unless it is recognised as internal
command (eg. @code{sleep}).
@item
Specify the maximum length of the result.
All queries that produce results, e.g. @code{SELECT}, @code{SHOW},
@code{EXPLAIN}, etc., must be preceded with @code{@@/path/to/result/file}. The
file must contain the expected results. An easy way to generate the result
file is to run @code{mysqltest -r < t/test-case-name.test} from
@code{mysql-test} directory, and then edit the generated result files, if
needed, to adjust them to the expected output. In that case, be very careful
about not adding or deleting any invisible characters - make sure to only
change the text and/or delete lines. If you have to insert a line, make sure
the fields are separated with a hard tab, and there is a hard tab at the end.
You may want to use @code{od -c} to make sure your text editor has not messed
anything up during edit. We, of course, hope that you will never have to edit
the output of @code{mysqltest -r} as you only have to do it when you find a
bug.
@item
Specify (for @code{REAL} functions) the maximum number of decimals.
To be consistent with our setup, you should put your result files in
@code{mysql-test/r} directory and name them @code{test_name.result}. If the
test produces more than one result, you should use @code{test_name.a.result},
@code{test_name.b.result}, etc.
@item
Specify whether or not the result can be @code{NULL}.
@end itemize
If a statement returns an error, you should on the line before the statement
specify with the @code{--error error-number}. The error number can be
a list of possible error numbers separated with @code{','}.
@item @code{xxx_deinit()} (optional)
The deinitialisation function for @code{xxx()}. It should deallocate any
memory allocated by the initialisation function.
@end table
When a SQL statement invokes @code{XXX()}, MySQL calls the
initialisation function @code{xxx_init()} to let it perform any required
setup, such as argument checking or memory allocation. If @code{xxx_init()}
returns an error, the SQL statement is aborted with an error message and the
main and deinitialisation functions are not called. Otherwise, the main
function @code{xxx()} is called once for each row. After all rows have been
processed, the deinitialisation function @code{xxx_deinit()} is called so it
can perform any required cleanup.
All functions must be thread safe (not just the main function,
but the initialisation and deinitialisation functions as well). This means
that you are not allowed to allocate any global or static variables that
change! If you need memory, you should allocate it in @code{xxx_init()}
and free it in @code{xxx_deinit()}.
@node UDF calling sequences, UDF arguments, Adding UDF, Adding UDF
@subsubsection UDF Calling Sequences
@cindex calling sequences, UDF
The main function should be declared as shown below. Note that the return
type and parameters differ, depending on whether you will declare the SQL
function @code{XXX()} to return @code{STRING}, @code{INTEGER}, or @code{REAL}
in the @code{CREATE FUNCTION} statement:
@noindent
For @code{STRING} functions:
@example
char *xxx(UDF_INIT *initid, UDF_ARGS *args,
char *result, unsigned long *length,
char *is_null, char *error);
@end example
@noindent
For @code{INTEGER} functions:
@example
long long xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
@end example
@noindent
For @code{REAL} functions:
@item
If you are writing a replication test case, you should on the first line of
the test file, put @code{source include/master-slave.inc;}. To switch between
master and slave, use @code{connection master;} and @code{connection slave;}.
If you need to do something on an alternate connection, you can do
@code{connection master1;} for the master, and @code{connection slave1;} for
the slave.
@item
If you need to do something in a loop, you can use something like this:
@example
double xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
let $1=1000;
while ($1)
@{
# do your queries here
dec $1;
@}
@end example
The initialisation and deinitialisation functions are declared like this:
@example
my_bool xxx_init(UDF_INIT *initid, UDF_ARGS *args, char *message);
@item
To sleep between queries, use the @code{sleep} command. It supports fractions
of a second, so you can do @code{sleep 1.3;}, for example, to sleep 1.3
seconds.
void xxx_deinit(UDF_INIT *initid);
@end example
@item
To run the slave with additional options for your test case, put them
in the command-line format in @code{mysql-test/t/test_name-slave.opt}. For
the master, put them in @code{mysql-test/t/test_name-master.opt}.
The @code{initid} parameter is passed to all three functions. It points to a
@code{UDF_INIT} structure that is used to communicate information between
functions. The @code{UDF_INIT} structure members are listed below. The
initialisation function should fill in any members that it wishes to change.
(To use the default for a member, leave it unchanged.):
@item
If you have a question about the test suite, or have a test case to contribute,
e-mail to @email{internals@@lists.mysql.com}. As the list does not accept
attachments, you should ftp all the relevant files to:
@uref{ftp://support.mysql.com/pub/mysql/Incoming}
@table @code
@item my_bool maybe_null
@code{xxx_init()} should set @code{maybe_null} to @code{1} if @code{xxx()}
can return @code{NULL}. The default value is @code{1} if any of the
arguments are declared @code{maybe_null}.
@end itemize
@item unsigned int decimals
Number of decimals. The default value is the maximum number of decimals in
the arguments passed to the main function. (For example, if the function is
passed @code{1.34}, @code{1.345}, and @code{1.3}, the default would be 3,
because @code{1.345} has 3 decimals.
@item unsigned int max_length
The maximum length of the string result. The default value differs depending
on the result type of the function. For string functions, the default is the
length of the longest argument. For integer functions, the default is 21
digits. For real functions, the default is 13 plus the number of decimals
indicated by @code{initid->decimals}. (For numeric functions, the length
includes any sign or decimal point characters.)
@node Reporting mysqltest bugs, , extending mysqltest, MySQL test suite
@subsubsection Reporting Bugs in the MySQL Test Suite
If you want to return a blob, you can set this to 65K or 16M; This
memory is not allocated but used to decide which column type to use if
there is a need to temporary store the data.
If your MySQL version doesn't pass the test suite you should
do the following:
@item char *ptr
A pointer that the function can use for its own purposes. For example,
functions can use @code{initid->ptr} to communicate allocated memory
between functions. In @code{xxx_init()}, allocate the memory and assign it
to this pointer:
@itemize @bullet
@item
Don't send a bug report before you have found out as much as possible of
what when wrong! When you do it, please use the @code{mysqlbug} script
so that we can get information about your system and @code{MySQL}
version. @xref{Bug reports}.
@item
Make sure to include the output of @code{mysql-test-run}, as well as
contents of all @code{.reject} files in @code{mysql-test/r} directory.
@item
If a test in the test suite fails, check if the test fails also when run
by its own:
@example
initid->ptr = allocated_memory;
cd mysql-test
mysql-test-run --local test-name
@end example
In @code{xxx()} and @code{xxx_deinit()}, refer to @code{initid->ptr} to use
or deallocate the memory.
@end table
If this fails, then you should configure MySQL with
@code{--with-debug} and run @code{mysql-test-run} with the
@code{--debug} option. If this also fails send the trace file
@file{var/tmp/master.trace} to ftp://support.mysql.com/pub/mysql/secret
so that we can examine it. Please remember to also include a full
description of your system, the version of the mysqld binary and how you
compiled it.
@node UDF arguments, UDF return values, UDF calling sequences, Adding UDF
@subsubsection Argument Processing
@item
Try also to run @code{mysql-test-run} with the @code{--force} option to
see if there is any other test that fails.
@cindex argument processing
@cindex processing, arguments
@item
If you have compiled MySQL yourself, check our manual for how
to compile MySQL on your platform or, preferable, use one of
the binaries we have compiled for you at
@uref{http://www.mysql.com/downloads/}. All our standard binaries should
pass the test suite !
The @code{args} parameter points to a @code{UDF_ARGS} structure that thas the
members listed below:
@item
If you get an error, like @code{Result length mismatch} or @code{Result
content mismatch} it means that the output of the test didn't match
exactly the expected output. This could be a bug in MySQL or
that your mysqld version produces slight different results under some
circumstances.
@table @code
@item unsigned int arg_count
The number of arguments. Check this value in the initialisation function
if you want your function to be called with a particular number of arguments.
For example:
Failed test results are put in a file with the same base name as the
result file with the @code{.reject} extension. If your test case is
failing, you should do a diff on the two files. If you cannot see how
they are different, examine both with @code{od -c} and also check their
lengths.
@example
if (args->arg_count != 2)
@{
strcpy(message,"XXX() requires two arguments");
return 1;
@}
@end example
@item
If a test fails totally, you should check the logs file in the
@code{mysql-test/var/log} directory for hints of what went wrong.
@item
If you have compiled MySQL with debugging you can try to debug this
by running @code{mysql-test-run} with the @code{--gdb} and/or @code{--debug}
options.
@xref{Making trace files}.
@item enum Item_result *arg_type
The types for each argument. The possible type values are
@code{STRING_RESULT}, @code{INT_RESULT}, and @code{REAL_RESULT}.
If you have not compiled MySQL for debugging you should probably
do that. Just specify the @code{--with-debug} options to @code{configure}!
@xref{Installing source}.
@end itemize
To make sure that arguments are of a given type and return an
error if they are not, check the @code{arg_type} array in the initialisation
function. For example:
@example
if (args->arg_type[0] != STRING_RESULT ||
args->arg_type[1] != INT_RESULT)
@{
strcpy(message,"XXX() requires a string and an integer");
return 1;
@}
@end example
@node Adding functions, Adding procedures, MySQL internals, Extending MySQL
@section Adding New Functions to MySQL
As an alternative to requiring your function's arguments to be of particular
types, you can use the initialisation function to set the @code{arg_type}
elements to the types you want. This causes MySQL to coerce
arguments to those types for each call to @code{xxx()}. For example, to
specify coercion of the first two arguments to string and integer, do this in
@code{xxx_init()}:
@cindex functions, new
@cindex adding, new functions
@cindex user-defined functions, adding
@cindex UDFs, defined
@cindex functions, user-defined
@example
args->arg_type[0] = STRING_RESULT;
args->arg_type[1] = INT_RESULT;
@end example
There are two ways to add new functions to MySQL:
@item char **args
@code{args->args} communicates information to the initialisation function
about the general nature of the arguments your function was called with. For a
constant argument @code{i}, @code{args->args[i]} points to the argument
value. (See below for instructions on how to access the value properly.)
For a non-constant argument, @code{args->args[i]} is @code{0}.
A constant argument is an expression that uses only constants, such as
@code{3} or @code{4*7-2} or @code{SIN(3.14)}. A non-constant argument is an
expression that refers to values that may change from row to row, such as
column names or functions that are called with non-constant arguments.
@itemize @bullet
@item You can add the function through the user-definable function (UDF)
interface. User-definable functions are added and removed dynamically using
the @code{CREATE FUNCTION} and @code{DROP FUNCTION} statements.
@xref{CREATE FUNCTION, , @code{CREATE FUNCTION}}.
For each invocation of the main function, @code{args->args} contains the
actual arguments that are passed for the row currently being processed.
@item You can add the function as a native (built in) MySQL function.
Native functions are compiled into the @code{mysqld} server and become
available on a permanent basis.
@end itemize
Functions can refer to an argument @code{i} as follows:
Each method has advantages and disadvantages:
@itemize @bullet
@item
An argument of type @code{STRING_RESULT} is given as a string pointer plus a
length, to allow handling of binary data or data of arbitrary length. The
string contents are available as @code{args->args[i]} and the string length
is @code{args->lengths[i]}. You should not assume that strings are
null-terminated.
If you write a user-definable function, you must install the object file
in addition to the server itself. If you compile your function into the
server, you don't need to do that.
@item
For an argument of type @code{INT_RESULT}, you must cast
@code{args->args[i]} to a @code{long long} value:
You can add UDFs to a binary MySQL distribution. Native functions
require you to modify a source distribution.
@item
If you upgrade your MySQL distribution, you can continue to use your
previously installed UDFs. For native functions, you must repeat your
modifications each time you upgrade.
@end itemize
@example
long long int_val;
int_val = *((long long*) args->args[i]);
@end example
Whichever method you use to add new functions, they may be used just like
native functions such as @code{ABS()} or @code{SOUNDEX()}.
@item
For an argument of type @code{REAL_RESULT}, you must cast
@code{args->args[i]} to a @code{double} value:
@menu
* CREATE FUNCTION:: @code{CREATE FUNCTION/DROP FUNCTION} Syntax
* Adding UDF:: Adding a new user-definable function
* Adding native function:: Adding a new native function
@end menu
@example
double real_val;
real_val = *((double*) args->args[i]);
@end example
@end itemize
@item unsigned long *lengths
For the initialisation function, the @code{lengths} array indicates the
maximum string length for each argument. For each invocation of the main
function, @code{lengths} contains the actual lengths of any string arguments
that are passed for the row currently being processed. For arguments of
types @code{INT_RESULT} or @code{REAL_RESULT}, @code{lengths} still contains
the maximum length of the argument (as for the initialisation function).
@end table
@node CREATE FUNCTION, Adding UDF, Adding functions, Adding functions
@subsection @code{CREATE FUNCTION/DROP FUNCTION} Syntax
@findex CREATE FUNCTION
@findex DROP FUNCTION
@findex UDF functions
@findex User-defined functions
@findex Functions, user-defined
@node UDF return values, UDF compiling, UDF arguments, Adding UDF
@subsubsection Return Values and Error Handling
@example
CREATE [AGGREGATE] FUNCTION function_name RETURNS @{STRING|REAL|INTEGER@}
SONAME shared_library_name
@cindex UDFs, return values
@cindex return values, UDFs
@cindex errors, handling for UDFs
@cindex handling, errors
DROP FUNCTION function_name
@end example
The initialisation function should return @code{0} if no error occurred and
@code{1} otherwise. If an error occurs, @code{xxx_init()} should store a
null-terminated error message in the @code{message} parameter. The message
will be returned to the client. The message buffer is
@code{MYSQL_ERRMSG_SIZE} characters long, but you should try to keep the
message to less than 80 characters so that it fits the width of a standard
terminal screen.
A user-definable function (UDF) is a way to extend MySQL with a new
function that works like native (built in) MySQL functions such as
@code{ABS()} and @code{CONCAT()}.
The return value of the main function @code{xxx()} is the function value, for
@code{long long} and @code{double} functions. A string functions should
return a pointer to the result and store the length of the string in the
@code{length} arguments.
@code{AGGREGATE} is a new option for MySQL Version 3.23. An
@code{AGGREGATE} function works exactly like a native MySQL
@code{GROUP} function like @code{SUM} or @code{COUNT()}.
Set these to the contents and length of the return value. For example:
@code{CREATE FUNCTION} saves the function's name, type, and shared library
name in the @code{mysql.func} system table. You must have the
@strong{insert} and @strong{delete} privileges for the @code{mysql} database
to create and drop functions.
@example
memcpy(result, "result string", 13);
*length = 13;
@end example
All active functions are reloaded each time the server starts, unless
you start @code{mysqld} with the @code{--skip-grant-tables} option. In
this case, UDF initialisation is skipped and UDFs are unavailable.
(An active function is one that has been loaded with @code{CREATE FUNCTION}
and not removed with @code{DROP FUNCTION}.)
The @code{result} buffer that is passed to the calc function is 255 byte
big. If your result fits in this, you don't have to worry about memory
allocation for results.
For instructions on writing user-definable functions, see @ref{Adding
functions}. For the UDF mechanism to work, functions must be written in C or
C++, your operating system must support dynamic loading and you must have
compiled @code{mysqld} dynamically (not statically).
If your string function needs to return a string longer than 255 bytes,
you must allocate the space for it with @code{malloc()} in your
@code{xxx_init()} function or your @code{xxx()} function and free it in
your @code{xxx_deinit()} function. You can store the allocated memory
in the @code{ptr} slot in the @code{UDF_INIT} structure for reuse by
future @code{xxx()} calls. @xref{UDF calling sequences}.
Note that to make @code{AGGREGATE} work, you must have a
@code{mysql.func} table that contains the column @code{type}. If this
is not the case, you should run the script
@code{mysql_fix_privilege_tables} to get this fixed.
To indicate a return value of @code{NULL} in the main function, set
@code{is_null} to @code{1}:
@example
*is_null = 1;
@end example
@node Adding UDF, Adding native function, CREATE FUNCTION, Adding functions
@subsection Adding a New User-definable Function
To indicate an error return in the main function, set the @code{error}
parameter to @code{1}:
@cindex adding, user-definable functions
@cindex user-defined functions, adding
@cindex functions, user-definable, adding
@example
*error = 1;
@end example
@menu
* UDF calling sequences:: UDF calling sequences
* UDF arguments:: Argument processing
* UDF return values:: Return values and error handling
* UDF compiling:: Compiling and installing user-definable functions
@end menu
If @code{xxx()} sets @code{*error} to @code{1} for any row, the function
value is @code{NULL} for the current row and for any subsequent rows
processed by the statement in which @code{XXX()} was invoked. (@code{xxx()}
will not even be called for subsequent rows.) @strong{NOTE:} In
MySQL versions prior to 3.22.10, you should set both @code{*error}
and @code{*is_null}:
@example
*error = 1;
*is_null = 1;
@end example
For the UDF mechanism to work, functions must be written in C or C++ and your
operating system must support dynamic loading. The MySQL source
distribution includes a file @file{sql/udf_example.cc} that defines 5 new
functions. Consult this file to see how UDF calling conventions work.
For @code{mysqld} to be able to use UDF functions, you should configure MySQL
with @code{--with-mysqld-ldflags=-rdynamic} The reason is that to on
many platforms (including Linux) you can load a dynamic library (with
@code{dlopen()}) from a static linked program, which you would get if
you are using @code{--with-mysqld-ldflags=-all-static} If you want to
use an UDF that needs to access symbols from @code{mysqld} (like the
@code{methaphone} example in @file{sql/udf_example.cc} that uses
@code{default_charset_info}), you must link the program with
@code{-rdynamic} (see @code{man dlopen}).
@node UDF compiling, , UDF return values, Adding UDF
@subsubsection Compiling and Installing User-definable Functions
For each function that you want to use in SQL statements, you should define
corresponding C (or C++) functions. In the discussion below, the name
``xxx'' is used for an example function name. To distinquish between SQL and
C/C++ usage, @code{XXX()} (uppercase) indicates a SQL function call, and
@code{xxx()} (lowercase) indicates a C/C++ function call.
@cindex compiling, user-defined functions
@cindex UDFs, compiling
@cindex installing, user-defined functions
The C/C++ functions that you write to implement the interface for
@code{XXX()} are:
Files implementing UDFs must be compiled and installed on the host where the
server runs. This process is described below for the example UDF file
@file{udf_example.cc} that is included in the MySQL source
distribution. This file contains the following functions:
@table @asis
@item @code{xxx()} (required)
The main function. This is where the function result is computed.
The correspondence between the SQL type and return type of your C/C++
function is shown below:
@multitable @columnfractions .2 .8
@item @strong{SQL type} @tab @strong{C/C++ type}
@item @code{STRING} @tab @code{char *}
@item @code{INTEGER} @tab @code{long long}
@item @code{REAL} @tab @code{double}
@end multitable
@item @code{xxx_init()} (optional)
The initialisation function for @code{xxx()}. It can be used to:
@itemize @bullet
@item
@code{metaphon()} returns a metaphon string of the string argument.
This is something like a soundex string, but it's more tuned for English.
Check the number of arguments to @code{XXX()}.
@item
@code{myfunc_double()} returns the sum of the ASCII values of the
characters in its arguments, divided by the sum of the length of its arguments.
Check that the arguments are of a required type or, alternatively,
tell MySQL to coerce arguments to the types you want when
the main function is called.
@item
@code{myfunc_int()} returns the sum of the length of its arguments.
Allocate any memory required by the main function.
@item
@code{sequence([const int])} returns an sequence starting from the given
number or 1 if no number has been given.
Specify the maximum length of the result.
@item
@code{lookup()} returns the IP number for a hostname.
Specify (for @code{REAL} functions) the maximum number of decimals.
@item
@code{reverse_lookup()} returns the hostname for an IP number.
The function may be called with a string @code{"xxx.xxx.xxx.xxx"} or
four numbers.
Specify whether or not the result can be @code{NULL}.
@end itemize
A dynamically loadable file should be compiled as a sharable object file,
using a command something like this:
@item @code{xxx_deinit()} (optional)
The deinitialisation function for @code{xxx()}. It should deallocate any
memory allocated by the initialisation function.
@end table
When a SQL statement invokes @code{XXX()}, MySQL calls the
initialisation function @code{xxx_init()} to let it perform any required
setup, such as argument checking or memory allocation. If @code{xxx_init()}
returns an error, the SQL statement is aborted with an error message and the
main and deinitialisation functions are not called. Otherwise, the main
function @code{xxx()} is called once for each row. After all rows have been
processed, the deinitialisation function @code{xxx_deinit()} is called so it
can perform any required cleanup.
All functions must be thread safe (not just the main function,
but the initialisation and deinitialisation functions as well). This means
that you are not allowed to allocate any global or static variables that
change! If you need memory, you should allocate it in @code{xxx_init()}
and free it in @code{xxx_deinit()}.
@node UDF calling sequences, UDF arguments, Adding UDF, Adding UDF
@subsubsection UDF Calling Sequences
@cindex calling sequences, UDF
The main function should be declared as shown below. Note that the return
type and parameters differ, depending on whether you will declare the SQL
function @code{XXX()} to return @code{STRING}, @code{INTEGER}, or @code{REAL}
in the @code{CREATE FUNCTION} statement:
@noindent
For @code{STRING} functions:
@example
shell> gcc -shared -o udf_example.so myfunc.cc
char *xxx(UDF_INIT *initid, UDF_ARGS *args,
char *result, unsigned long *length,
char *is_null, char *error);
@end example
You can easily find out the correct compiler options for your system by
running this command in the @file{sql} directory of your MySQL
source tree:
@noindent
For @code{INTEGER} functions:
@example
shell> make udf_example.o
long long xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
@end example
You should run a compile command similar to the one that @code{make} displays,
except that you should remove the @code{-c} option near the end of the line
and add @code{-o udf_example.so} to the end of the line. (On some systems,
you may need to leave the @code{-c} on the command.)
@noindent
For @code{REAL} functions:
Once you compile a shared object containing UDFs, you must install it
and tell MySQL about it. Compiling a shared object from
@file{udf_example.cc} produces a file named something like
@file{udf_example.so} (the exact name may vary from platform to platform).
Copy this file to some directory searched by @code{ld}, such as
@file{/usr/lib}. On many systems, you can set the @code{LD_LIBRARY} or
@code{LD_LIBRARY_PATH} environment variable to point at the directory where
you have your UDF function files. The @code{dlopen} manual page tells you
which variable you should use on your system. You should set this in
@code{mysql.server} or @code{safe_mysqld} and restart @code{mysqld}.
@example
double xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
@end example
After the library is installed, notify @code{mysqld} about the new
functions with these commands:
The initialisation and deinitialisation functions are declared like this:
@example
mysql> CREATE FUNCTION metaphon RETURNS STRING SONAME "udf_example.so";
mysql> CREATE FUNCTION myfunc_double RETURNS REAL SONAME "udf_example.so";
mysql> CREATE FUNCTION myfunc_int RETURNS INTEGER SONAME "udf_example.so";
mysql> CREATE FUNCTION lookup RETURNS STRING SONAME "udf_example.so";
mysql> CREATE FUNCTION reverse_lookup RETURNS STRING SONAME "udf_example.so";
mysql> CREATE AGGREGATE FUNCTION avgcost RETURNS REAL SONAME "udf_example.so";
my_bool xxx_init(UDF_INIT *initid, UDF_ARGS *args, char *message);
void xxx_deinit(UDF_INIT *initid);
@end example
Functions can be deleted using @code{DROP FUNCTION}:
The @code{initid} parameter is passed to all three functions. It points to a
@code{UDF_INIT} structure that is used to communicate information between
functions. The @code{UDF_INIT} structure members are listed below. The
initialisation function should fill in any members that it wishes to change.
(To use the default for a member, leave it unchanged.):
@table @code
@item my_bool maybe_null
@code{xxx_init()} should set @code{maybe_null} to @code{1} if @code{xxx()}
can return @code{NULL}. The default value is @code{1} if any of the
arguments are declared @code{maybe_null}.
@item unsigned int decimals
Number of decimals. The default value is the maximum number of decimals in
the arguments passed to the main function. (For example, if the function is
passed @code{1.34}, @code{1.345}, and @code{1.3}, the default would be 3,
because @code{1.345} has 3 decimals.
@item unsigned int max_length
The maximum length of the string result. The default value differs depending
on the result type of the function. For string functions, the default is the
length of the longest argument. For integer functions, the default is 21
digits. For real functions, the default is 13 plus the number of decimals
indicated by @code{initid->decimals}. (For numeric functions, the length
includes any sign or decimal point characters.)
If you want to return a blob, you can set this to 65K or 16M; This
memory is not allocated but used to decide which column type to use if
there is a need to temporary store the data.
@item char *ptr
A pointer that the function can use for its own purposes. For example,
functions can use @code{initid->ptr} to communicate allocated memory
between functions. In @code{xxx_init()}, allocate the memory and assign it
to this pointer:
@example
mysql> DROP FUNCTION metaphon;
mysql> DROP FUNCTION myfunc_double;
mysql> DROP FUNCTION myfunc_int;
mysql> DROP FUNCTION lookup;
mysql> DROP FUNCTION reverse_lookup;
mysql> DROP FUNCTION avgcost;
initid->ptr = allocated_memory;
@end example
The @code{CREATE FUNCTION} and @code{DROP FUNCTION} statements update the
system table @code{func} in the @code{mysql} database. The function's name,
type and shared library name are saved in the table. You must have the
@strong{insert} and @strong{delete} privileges for the @code{mysql} database
to create and drop functions.
In @code{xxx()} and @code{xxx_deinit()}, refer to @code{initid->ptr} to use
or deallocate the memory.
@end table
You should not use @code{CREATE FUNCTION} to add a function that has already
been created. If you need to reinstall a function, you should remove it with
@code{DROP FUNCTION} and then reinstall it with @code{CREATE FUNCTION}. You
would need to do this, for example, if you recompile a new version of your
function, so that @code{mysqld} gets the new version. Otherwise the server
will continue to use the old version.
Active functions are reloaded each time the server starts, unless you start
@code{mysqld} with the @code{--skip-grant-tables} option. In this case, UDF
initialisation is skipped and UDFs are unavailable. (An active function is
one that has been loaded with @code{CREATE FUNCTION} and not removed with
@code{DROP FUNCTION}.)
@node UDF arguments, UDF return values, UDF calling sequences, Adding UDF
@subsubsection Argument Processing
@cindex argument processing
@cindex processing, arguments
@node Adding native function, , Adding UDF, Adding functions
@subsection Adding a New Native Function
The @code{args} parameter points to a @code{UDF_ARGS} structure that thas the
members listed below:
@cindex adding, native functions
@cindex native functions, adding
@cindex functions, native, adding
@table @code
@item unsigned int arg_count
The number of arguments. Check this value in the initialisation function
if you want your function to be called with a particular number of arguments.
For example:
The procedure for adding a new native function is described below. Note
that you cannot add native functions to a binary distribution because
the procedure involves modifying MySQL source code. You must
compile MySQL yourself from a source distribution. Also note
that if you migrate to another version of MySQL (for example,
when a new version is released), you will need to repeat the procedure
with the new version.
@example
if (args->arg_count != 2)
@{
strcpy(message,"XXX() requires two arguments");
return 1;
@}
@end example
To add a new native MySQL function, follow these steps:
@enumerate
@item
Add one line to @file{lex.h} that defines the function name in the
@code{sql_functions[]} array.
@item
If the function prototype is simple (just takes zero, one, two or three
arguments), you should in lex.h specify SYM(FUNC_ARG#) (where # is the
number of arguments) as the second argument in the
@code{sql_functions[]} array and add a function that creates a function
object in @file{item_create.cc}. Take a look at @code{"ABS"} and
@code{create_funcs_abs()} for an example of this.
@item enum Item_result *arg_type
The types for each argument. The possible type values are
@code{STRING_RESULT}, @code{INT_RESULT}, and @code{REAL_RESULT}.
To make sure that arguments are of a given type and return an
error if they are not, check the @code{arg_type} array in the initialisation
function. For example:
If the function prototype is complicated (for example takes a variable number
of arguments), you should add two lines to @file{sql_yacc.yy}. One
indicates the preprocessor symbol that @code{yacc} should define (this
should be added at the beginning of the file). Then define the function
parameters and add an ``item'' with these parameters to the
@code{simple_expr} parsing rule. For an example, check all occurrences
of @code{ATAN} in @file{sql_yacc.yy} to see how this is done.
@item
In @file{item_func.h}, declare a class inheriting from @code{Item_num_func} or
@code{Item_str_func}, depending on whether your function returns a number or a
string.
@item
In @file{item_func.cc}, add one of the following declarations, depending
on whether you are defining a numeric or string function:
@example
double Item_func_newname::val()
longlong Item_func_newname::val_int()
String *Item_func_newname::Str(String *str)
if (args->arg_type[0] != STRING_RESULT ||
args->arg_type[1] != INT_RESULT)
@{
strcpy(message,"XXX() requires a string and an integer");
return 1;
@}
@end example
If you inherit your object from any of the standard items (like
@code{Item_num_func} you probably only have to define one of the above
functions and let the parent object take care of the other functions.
For example, the @code{Item_str_func} class defines a @code{val()} function
that executes @code{atof()} on the value returned by @code{::str()}.
As an alternative to requiring your function's arguments to be of particular
types, you can use the initialisation function to set the @code{arg_type}
elements to the types you want. This causes MySQL to coerce
arguments to those types for each call to @code{xxx()}. For example, to
specify coercion of the first two arguments to string and integer, do this in
@code{xxx_init()}:
@item
You should probably also define the following object function:
@example
void Item_func_newname::fix_length_and_dec()
args->arg_type[0] = STRING_RESULT;
args->arg_type[1] = INT_RESULT;
@end example
This function should at least calculate @code{max_length} based on the
given arguments. @code{max_length} is the maximum number of characters
the function may return. This function should also set @code{maybe_null
= 0} if the main function can't return a @code{NULL} value. The
function can check if any of the function arguments can return
@code{NULL} by checking the arguments @code{maybe_null} variable. You
can take a look at @code{Item_func_mod::fix_length_and_dec} for a
typical example of how to do this.
@end enumerate
All functions must be thread safe (in other words, don't use any global or
static variables in the functions without protecting them with mutexes).
@item char **args
@code{args->args} communicates information to the initialisation function
about the general nature of the arguments your function was called with. For a
constant argument @code{i}, @code{args->args[i]} points to the argument
value. (See below for instructions on how to access the value properly.)
For a non-constant argument, @code{args->args[i]} is @code{0}.
A constant argument is an expression that uses only constants, such as
@code{3} or @code{4*7-2} or @code{SIN(3.14)}. A non-constant argument is an
expression that refers to values that may change from row to row, such as
column names or functions that are called with non-constant arguments.
If you want to return @code{NULL}, from @code{::val()}, @code{::val_int()}
or @code{::str()} you should set @code{null_value} to 1 and return 0.
For each invocation of the main function, @code{args->args} contains the
actual arguments that are passed for the row currently being processed.
For @code{::str()} object functions, there are some additional
considerations to be aware of:
Functions can refer to an argument @code{i} as follows:
@itemize @bullet
@item
The @code{String *str} argument provides a string buffer that may be
used to hold the result. (For more information about the @code{String} type,
take a look at the @file{sql_string.h} file.)
@item
The @code{::str()} function should return the string that holds the result or
@code{(char*) 0} if the result is @code{NULL}.
@item
All current string functions try to avoid allocating any memory unless
absolutely necessary!
@end itemize
An argument of type @code{STRING_RESULT} is given as a string pointer plus a
length, to allow handling of binary data or data of arbitrary length. The
string contents are available as @code{args->args[i]} and the string length
is @code{args->lengths[i]}. You should not assume that strings are
null-terminated.
@node Adding procedures, MySQL internals, Adding functions, Extending MySQL
@section Adding New Procedures to MySQL
@item
For an argument of type @code{INT_RESULT}, you must cast
@code{args->args[i]} to a @code{long long} value:
@cindex procedures, adding
@cindex adding, procedures
@cindex new procedures, adding
@example
long long int_val;
int_val = *((long long*) args->args[i]);
@end example
In MySQL, you can define a procedure in C++ that can access and
modify the data in a query before it is sent to the client. The modification
can be done on row-by-row or @code{GROUP BY} level.
@item
For an argument of type @code{REAL_RESULT}, you must cast
@code{args->args[i]} to a @code{double} value:
We have created an example procedure in MySQL Version 3.23 to
show you what can be done.
@example
double real_val;
real_val = *((double*) args->args[i]);
@end example
@end itemize
Additionally we recommend you to take a look at 'mylua', which you can find in the Contrib directory. @xref{Contrib}. Which this you can use the LUA
language to load a procedure at runtime into @code{mysqld}.
@item unsigned long *lengths
For the initialisation function, the @code{lengths} array indicates the
maximum string length for each argument. For each invocation of the main
function, @code{lengths} contains the actual lengths of any string arguments
that are passed for the row currently being processed. For arguments of
types @code{INT_RESULT} or @code{REAL_RESULT}, @code{lengths} still contains
the maximum length of the argument (as for the initialisation function).
@end table
@menu
* procedure analyse:: Procedure analyse
* Writing a procedure:: Writing a procedure.
@end menu
@node UDF return values, UDF compiling, UDF arguments, Adding UDF
@subsubsection Return Values and Error Handling
@node procedure analyse, Writing a procedure, Adding procedures, Adding procedures
@subsection Procedure Analyse
@cindex UDFs, return values
@cindex return values, UDFs
@cindex errors, handling for UDFs
@cindex handling, errors
@code{analyse([max elements,[max memory]])}
The initialisation function should return @code{0} if no error occurred and
@code{1} otherwise. If an error occurs, @code{xxx_init()} should store a
null-terminated error message in the @code{message} parameter. The message
will be returned to the client. The message buffer is
@code{MYSQL_ERRMSG_SIZE} characters long, but you should try to keep the
message to less than 80 characters so that it fits the width of a standard
terminal screen.
This procedure is defined in the @file{sql/sql_analyse.cc}. This
examines the result from your query and returns an analysis of the
results:
The return value of the main function @code{xxx()} is the function value, for
@code{long long} and @code{double} functions. A string functions should
return a pointer to the result and store the length of the string in the
@code{length} arguments.
@itemize @bullet
@item
@code{max elements} (default 256) is the maximum number of distinct values
@code{analyse} will notice per column. This is used by @code{analyse} to check if
the optimal column type should be of type @code{ENUM}.
@item
@code{max memory} (default 8192) is the maximum memory @code{analyse} should
allocate per column while trying to find all distinct values.
@end itemize
Set these to the contents and length of the return value. For example:
@example
SELECT ... FROM ... WHERE ... PROCEDURE ANALYSE([max elements,[max memory]])
memcpy(result, "result string", 13);
*length = 13;
@end example
The @code{result} buffer that is passed to the calc function is 255 byte
big. If your result fits in this, you don't have to worry about memory
allocation for results.
@node Writing a procedure, , procedure analyse, Adding procedures
@subsection Writing a Procedure
For the moment, the only documentation for this is the source.
If your string function needs to return a string longer than 255 bytes,
you must allocate the space for it with @code{malloc()} in your
@code{xxx_init()} function or your @code{xxx()} function and free it in
your @code{xxx_deinit()} function. You can store the allocated memory
in the @code{ptr} slot in the @code{UDF_INIT} structure for reuse by
future @code{xxx()} calls. @xref{UDF calling sequences}.
You can find all information about procedures by examining the following files:
To indicate a return value of @code{NULL} in the main function, set
@code{is_null} to @code{1}:
@itemize @bullet
@item @file{sql/sql_analyse.cc}
@item @file{sql/procedure.h}
@item @file{sql/procedure.cc}
@item @file{sql/sql_select.cc}
@end itemize
@example
*is_null = 1;
@end example
To indicate an error return in the main function, set the @code{error}
parameter to @code{1}:
@node MySQL internals, , Adding procedures, Extending MySQL
@section MySQL Internals
@example
*error = 1;
@end example
@cindex internals
@cindex threads
If @code{xxx()} sets @code{*error} to @code{1} for any row, the function
value is @code{NULL} for the current row and for any subsequent rows
processed by the statement in which @code{XXX()} was invoked. (@code{xxx()}
will not even be called for subsequent rows.) @strong{NOTE:} In
MySQL versions prior to 3.22.10, you should set both @code{*error}
and @code{*is_null}:
This chapter describes a lot of things that you need to know when
working on the MySQL code. If you plan to contribute to MySQL
development, want to have access to the bleeding-edge in-between
versions code, or just want to keep track of development, follow the
instructions in @xref{Installing source tree}. If you are interested
in MySQL internals, you should also subscribe to our @code{internals}
mailing list. This list is relatively low traffic. For details on how
to subscribe, please see @ref{Mailing-list}.
@example
*error = 1;
*is_null = 1;
@end example
@menu
* MySQL threads:: MySQL threads
* MySQL test suite:: MySQL test suite
@end menu
@node UDF compiling, , UDF return values, Adding UDF
@subsubsection Compiling and Installing User-definable Functions
@node MySQL threads, MySQL test suite, MySQL internals, MySQL internals
@subsection MySQL Threads
@cindex compiling, user-defined functions
@cindex UDFs, compiling
@cindex installing, user-defined functions
The MySQL server creates the following threads:
Files implementing UDFs must be compiled and installed on the host where the
server runs. This process is described below for the example UDF file
@file{udf_example.cc} that is included in the MySQL source
distribution. This file contains the following functions:
@itemize @bullet
@item
The TCP/IP connection thread handles all connection requests and
creates a new dedicated thread to handle the authentication and
and SQL query processing for each connection.
@item
On Windows NT there is a named pipe handler thread that does the same work as
the TCP/IP connection thread on named pipe connect requests.
@item
The signal thread handles all signals. This thread also normally handles
alarms and calls @code{process_alarm()} to force timeouts on connections
that have been idle too long.
@code{metaphon()} returns a metaphon string of the string argument.
This is something like a soundex string, but it's more tuned for English.
@item
If @code{mysqld} is compiled with @code{-DUSE_ALARM_THREAD}, a dedicated
thread that handles alarms is created. This is only used on some systems where
there are problems with @code{sigwait()} or if one wants to use the
@code{thr_alarm()} code in ones application without a dedicated signal
handling thread.
@code{myfunc_double()} returns the sum of the ASCII values of the
characters in its arguments, divided by the sum of the length of its arguments.
@item
If one uses the @code{--flush_time=#} option, a dedicated thread is created
to flush all tables at the given interval.
@code{myfunc_int()} returns the sum of the length of its arguments.
@item
Every connection has its own thread.
@code{sequence([const int])} returns an sequence starting from the given
number or 1 if no number has been given.
@item
Every different table on which one uses @code{INSERT DELAYED} gets its
own thread.
@code{lookup()} returns the IP number for a hostname.
@item
If you use @code{--master-host}, a slave replication thread will be
started to read and apply updates from the master.
@code{reverse_lookup()} returns the hostname for an IP number.
The function may be called with a string @code{"xxx.xxx.xxx.xxx"} or
four numbers.
@end itemize
@code{mysqladmin processlist} only shows the connection, @code{INSERT DELAYED},
and replication threads.
@node MySQL test suite, , MySQL threads, MySQL internals
@subsection MySQL Test Suite
A dynamically loadable file should be compiled as a sharable object file,
using a command something like this:
@cindex mysqltest, MySQL Test Suite
@cindex testing mysqld, mysqltest
@example
shell> gcc -shared -o udf_example.so myfunc.cc
@end example
Until recently, our main full-coverage test suite was based on proprietary
customer data and for that reason has not been publicly available. The only
publicly available part of our testing process consisted of the @code{crash-me}
test, a Perl DBI/DBD benchmark found in the @code{sql-bench} directory, and
miscellaneous tests located in @code{tests} directory. The lack of a
standardised publicly available test suite has made it difficult for our users,
as well developers, to do regression tests on the MySQL code. To
address this problem, we have created a new test system that is included in
the source and binary distributions starting in Version 3.23.29.
You can easily find out the correct compiler options for your system by
running this command in the @file{sql} directory of your MySQL
source tree:
The current set of test cases doesn't test everything in MySQL, but it
should catch most obvious bugs in the SQL processing code, OS/library
issues, and is quite thorough in testing replication. Our eventual goal
is to have the tests cover 100% of the code. We welcome contributions
to our test suite. You may especially want to contribute tests that
examine the functionality critical to your system, as this will ensure
that all future MySQL releases will work well with your
applications.
@example
shell> make udf_example.o
@end example
@menu
* running mysqltest:: Running the MySQL Test Suite
* extending mysqltest:: Extending the MySQL Test Suite
* Reporting mysqltest bugs:: Reporting Bugs in the MySQL Test Suite
@end menu
You should run a compile command similar to the one that @code{make} displays,
except that you should remove the @code{-c} option near the end of the line
and add @code{-o udf_example.so} to the end of the line. (On some systems,
you may need to leave the @code{-c} on the command.)
Once you compile a shared object containing UDFs, you must install it
and tell MySQL about it. Compiling a shared object from
@file{udf_example.cc} produces a file named something like
@file{udf_example.so} (the exact name may vary from platform to platform).
Copy this file to some directory searched by @code{ld}, such as
@file{/usr/lib}. On many systems, you can set the @code{LD_LIBRARY} or
@code{LD_LIBRARY_PATH} environment variable to point at the directory where
you have your UDF function files. The @code{dlopen} manual page tells you
which variable you should use on your system. You should set this in
@code{mysql.server} or @code{safe_mysqld} and restart @code{mysqld}.
@node running mysqltest, extending mysqltest, MySQL test suite, MySQL test suite
@subsubsection Running the MySQL Test Suite
After the library is installed, notify @code{mysqld} about the new
functions with these commands:
The test system consist of a test language interpreter
(@code{mysqltest}), a shell script to run all
tests(@code{mysql-test-run}), the actual test cases written in a special
test language, and their expected results. To run the test suite on
your system after a build, type @code{make test} or
@code{mysql-test/mysql-test-run} from the source root. If you have
installed a binary distribution, @code{cd} to the install root
(eg. @code{/usr/local/mysql}), and do @code{scripts/mysql-test-run}.
All tests should succeed. If not, you should try to find out why and
report the problem if this is a bug in MySQL.
@xref{Reporting mysqltest bugs}.
@example
mysql> CREATE FUNCTION metaphon RETURNS STRING SONAME "udf_example.so";
mysql> CREATE FUNCTION myfunc_double RETURNS REAL SONAME "udf_example.so";
mysql> CREATE FUNCTION myfunc_int RETURNS INTEGER SONAME "udf_example.so";
mysql> CREATE FUNCTION lookup RETURNS STRING SONAME "udf_example.so";
mysql> CREATE FUNCTION reverse_lookup RETURNS STRING SONAME "udf_example.so";
mysql> CREATE AGGREGATE FUNCTION avgcost RETURNS REAL SONAME "udf_example.so";
@end example
If you have a copy of @code{mysqld} running on the machine where you want to
run the test suite you do not have to stop it, as long as it is not using
ports @code{9306} and @code{9307}. If one of those ports is taken, you should
edit @code{mysql-test-run} and change the values of the master and/or slave
port to one that is available.
Functions can be deleted using @code{DROP FUNCTION}:
You can run one individual test case with
@code{mysql-test/mysql-test-run test_name}.
@example
mysql> DROP FUNCTION metaphon;
mysql> DROP FUNCTION myfunc_double;
mysql> DROP FUNCTION myfunc_int;
mysql> DROP FUNCTION lookup;
mysql> DROP FUNCTION reverse_lookup;
mysql> DROP FUNCTION avgcost;
@end example
If one test fails, you should test running @code{mysql-test-run} with
the @code{--force} option to check if any other tests fails.
The @code{CREATE FUNCTION} and @code{DROP FUNCTION} statements update the
system table @code{func} in the @code{mysql} database. The function's name,
type and shared library name are saved in the table. You must have the
@strong{insert} and @strong{delete} privileges for the @code{mysql} database
to create and drop functions.
You should not use @code{CREATE FUNCTION} to add a function that has already
been created. If you need to reinstall a function, you should remove it with
@code{DROP FUNCTION} and then reinstall it with @code{CREATE FUNCTION}. You
would need to do this, for example, if you recompile a new version of your
function, so that @code{mysqld} gets the new version. Otherwise the server
will continue to use the old version.
@node extending mysqltest, Reporting mysqltest bugs, running mysqltest, MySQL test suite
@subsubsection Extending the MySQL Test Suite
Active functions are reloaded each time the server starts, unless you start
@code{mysqld} with the @code{--skip-grant-tables} option. In this case, UDF
initialisation is skipped and UDFs are unavailable. (An active function is
one that has been loaded with @code{CREATE FUNCTION} and not removed with
@code{DROP FUNCTION}.)
You can use the @code{mysqltest} language to write your own test cases.
Unfortunately, we have not yet written full documentation for it - we plan to
do this shortly. You can, however, look at our current test cases and use
them as an example. The following points should help you get started:
@itemize @bullet
@item
The tests are located in @code{mysql-test/t/*.test}
@node Adding native function, , Adding UDF, Adding functions
@subsection Adding a New Native Function
@item
A test case consists of @code{;} terminated statements and is similar to the
input of @code{mysql} command line client. A statement by default is a query
to be sent to MySQL server, unless it is recognised as internal
command (eg. @code{sleep}).
@cindex adding, native functions
@cindex native functions, adding
@cindex functions, native, adding
@item
All queries that produce results, e.g. @code{SELECT}, @code{SHOW},
@code{EXPLAIN}, etc., must be preceded with @code{@@/path/to/result/file}. The
file must contain the expected results. An easy way to generate the result
file is to run @code{mysqltest -r < t/test-case-name.test} from
@code{mysql-test} directory, and then edit the generated result files, if
needed, to adjust them to the expected output. In that case, be very careful
about not adding or deleting any invisible characters - make sure to only
change the text and/or delete lines. If you have to insert a line, make sure
the fields are separated with a hard tab, and there is a hard tab at the end.
You may want to use @code{od -c} to make sure your text editor has not messed
anything up during edit. We, of course, hope that you will never have to edit
the output of @code{mysqltest -r} as you only have to do it when you find a
bug.
The procedure for adding a new native function is described below. Note
that you cannot add native functions to a binary distribution because
the procedure involves modifying MySQL source code. You must
compile MySQL yourself from a source distribution. Also note
that if you migrate to another version of MySQL (for example,
when a new version is released), you will need to repeat the procedure
with the new version.
@item
To be consistent with our setup, you should put your result files in
@code{mysql-test/r} directory and name them @code{test_name.result}. If the
test produces more than one result, you should use @code{test_name.a.result},
@code{test_name.b.result}, etc.
To add a new native MySQL function, follow these steps:
@enumerate
@item
If a statement returns an error, you should on the line before the statement
specify with the @code{--error error-number}. The error number can be
a list of possible error numbers separated with @code{','}.
Add one line to @file{lex.h} that defines the function name in the
@code{sql_functions[]} array.
@item
If you are writing a replication test case, you should on the first line of
the test file, put @code{source include/master-slave.inc;}. To switch between
master and slave, use @code{connection master;} and @code{connection slave;}.
If you need to do something on an alternate connection, you can do
@code{connection master1;} for the master, and @code{connection slave1;} for
the slave.
If the function prototype is simple (just takes zero, one, two or three
arguments), you should in lex.h specify SYM(FUNC_ARG#) (where # is the
number of arguments) as the second argument in the
@code{sql_functions[]} array and add a function that creates a function
object in @file{item_create.cc}. Take a look at @code{"ABS"} and
@code{create_funcs_abs()} for an example of this.
If the function prototype is complicated (for example takes a variable number
of arguments), you should add two lines to @file{sql_yacc.yy}. One
indicates the preprocessor symbol that @code{yacc} should define (this
should be added at the beginning of the file). Then define the function
parameters and add an ``item'' with these parameters to the
@code{simple_expr} parsing rule. For an example, check all occurrences
of @code{ATAN} in @file{sql_yacc.yy} to see how this is done.
@item
If you need to do something in a loop, you can use something like this:
In @file{item_func.h}, declare a class inheriting from @code{Item_num_func} or
@code{Item_str_func}, depending on whether your function returns a number or a
string.
@item
In @file{item_func.cc}, add one of the following declarations, depending
on whether you are defining a numeric or string function:
@example
let $1=1000;
while ($1)
@{
# do your queries here
dec $1;
@}
double Item_func_newname::val()
longlong Item_func_newname::val_int()
String *Item_func_newname::Str(String *str)
@end example
@item
To sleep between queries, use the @code{sleep} command. It supports fractions
of a second, so you can do @code{sleep 1.3;}, for example, to sleep 1.3
seconds.
@item
To run the slave with additional options for your test case, put them
in the command-line format in @code{mysql-test/t/test_name-slave.opt}. For
the master, put them in @code{mysql-test/t/test_name-master.opt}.
If you inherit your object from any of the standard items (like
@code{Item_num_func} you probably only have to define one of the above
functions and let the parent object take care of the other functions.
For example, the @code{Item_str_func} class defines a @code{val()} function
that executes @code{atof()} on the value returned by @code{::str()}.
@item
If you have a question about the test suite, or have a test case to contribute,
e-mail to @email{internals@@lists.mysql.com}. As the list does not accept
attachments, you should ftp all the relevant files to:
@uref{ftp://support.mysql.com/pub/mysql/Incoming}
@end itemize
You should probably also define the following object function:
@example
void Item_func_newname::fix_length_and_dec()
@end example
This function should at least calculate @code{max_length} based on the
given arguments. @code{max_length} is the maximum number of characters
the function may return. This function should also set @code{maybe_null
= 0} if the main function can't return a @code{NULL} value. The
function can check if any of the function arguments can return
@code{NULL} by checking the arguments @code{maybe_null} variable. You
can take a look at @code{Item_func_mod::fix_length_and_dec} for a
typical example of how to do this.
@end enumerate
All functions must be thread safe (in other words, don't use any global or
static variables in the functions without protecting them with mutexes).
@node Reporting mysqltest bugs, , extending mysqltest, MySQL test suite
@subsubsection Reporting Bugs in the MySQL Test Suite
If you want to return @code{NULL}, from @code{::val()}, @code{::val_int()}
or @code{::str()} you should set @code{null_value} to 1 and return 0.
If your MySQL version doesn't pass the test suite you should
do the following:
For @code{::str()} object functions, there are some additional
considerations to be aware of:
@itemize @bullet
@item
Don't send a bug report before you have found out as much as possible of
what when wrong! When you do it, please use the @code{mysqlbug} script
so that we can get information about your system and @code{MySQL}
version. @xref{Bug reports}.
The @code{String *str} argument provides a string buffer that may be
used to hold the result. (For more information about the @code{String} type,
take a look at the @file{sql_string.h} file.)
@item
Make sure to include the output of @code{mysql-test-run}, as well as
contents of all @code{.reject} files in @code{mysql-test/r} directory.
The @code{::str()} function should return the string that holds the result or
@code{(char*) 0} if the result is @code{NULL}.
@item
If a test in the test suite fails, check if the test fails also when run
by its own:
All current string functions try to avoid allocating any memory unless
absolutely necessary!
@end itemize
@example
cd mysql-test
mysql-test-run --local test-name
@end example
If this fails, then you should configure MySQL with
@code{--with-debug} and run @code{mysql-test-run} with the
@code{--debug} option. If this also fails send the trace file
@file{var/tmp/master.trace} to ftp://support.mysql.com/pub/mysql/secret
so that we can examine it. Please remember to also include a full
description of your system, the version of the mysqld binary and how you
compiled it.
@node Adding procedures, , Adding functions, Extending MySQL
@section Adding New Procedures to MySQL
@item
Try also to run @code{mysql-test-run} with the @code{--force} option to
see if there is any other test that fails.
@cindex procedures, adding
@cindex adding, procedures
@cindex new procedures, adding
@item
If you have compiled MySQL yourself, check our manual for how
to compile MySQL on your platform or, preferable, use one of
the binaries we have compiled for you at
@uref{http://www.mysql.com/downloads/}. All our standard binaries should
pass the test suite !
In MySQL, you can define a procedure in C++ that can access and
modify the data in a query before it is sent to the client. The modification
can be done on row-by-row or @code{GROUP BY} level.
@item
If you get an error, like @code{Result length mismatch} or @code{Result
content mismatch} it means that the output of the test didn't match
exactly the expected output. This could be a bug in MySQL or
that your mysqld version produces slight different results under some
circumstances.
We have created an example procedure in MySQL Version 3.23 to
show you what can be done.
Failed test results are put in a file with the same base name as the
result file with the @code{.reject} extension. If your test case is
failing, you should do a diff on the two files. If you cannot see how
they are different, examine both with @code{od -c} and also check their
lengths.
Additionally we recommend you to take a look at 'mylua', which you can find in the Contrib directory. @xref{Contrib}. Which this you can use the LUA
language to load a procedure at runtime into @code{mysqld}.
@item
If a test fails totally, you should check the logs file in the
@code{mysql-test/var/log} directory for hints of what went wrong.
@menu
* procedure analyse:: Procedure analyse
* Writing a procedure:: Writing a procedure.
@end menu
@item
If you have compiled MySQL with debugging you can try to debug this
by running @code{mysql-test-run} with the @code{--gdb} and/or @code{--debug}
options.
@xref{Making trace files}.
If you have not compiled MySQL for debugging you should probably
do that. Just specify the @code{--with-debug} options to @code{configure}!
@xref{Installing source}.
@node procedure analyse, Writing a procedure, Adding procedures, Adding procedures
@subsection Procedure Analyse
@code{analyse([max elements,[max memory]])}
This procedure is defined in the @file{sql/sql_analyse.cc}. This
examines the result from your query and returns an analysis of the
results:
@itemize @bullet
@item
@code{max elements} (default 256) is the maximum number of distinct values
@code{analyse} will notice per column. This is used by @code{analyse} to check if
the optimal column type should be of type @code{ENUM}.
@item
@code{max memory} (default 8192) is the maximum memory @code{analyse} should
allocate per column while trying to find all distinct values.
@end itemize
@example
SELECT ... FROM ... WHERE ... PROCEDURE ANALYSE([max elements,[max memory]])
@end example
@node Writing a procedure, , procedure analyse, Adding procedures
@subsection Writing a Procedure
For the moment, the only documentation for this is the source.
You can find all information about procedures by examining the following files:
@itemize @bullet
@item @file{sql/sql_analyse.cc}
@item @file{sql/procedure.h}
@item @file{sql/procedure.cc}
@item @file{sql/sql_select.cc}
@end itemize
@node Problems, Users, Extending MySQL, Top
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7