Commit 51cad3f4 authored by Vincent Pelletier's avatar Vincent Pelletier

Major SQLCatalog rework on query-generation (not on indexation):

- further extend SearchKey concept
- define, implement and validate Interfaces
- add a column mapper (mapping is decided on a completely-formed query tree instead of being done individualy on each used column)
- state what is present for backward compatibility, warn about deprecated/dangerous uses


git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@25706 20353a03-c40f-0410-a6d1-a30d3c3de9de
parent 68a0d3c2
This diff is collapsed.
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class INode(Interface):
"""
Any kind of node in an Abstract Syntax Tree.
"""
def isLeaf():
"""
Returns True if current node is a leaf in node tree.
Returns False otherwise.
"""
def isColumn():
"""
Returns True if current node is a column in node tree.
Returns False otherwise.
"""
class IValueNode(INode):
"""
Value- and comparison-operator-containig node.
They are leaf nodes in the syntax tree.
"""
def getValue():
"""
Returns node's value.
"""
def getComparisonOperator():
"""
Returns node's comparison operator.
"""
class ILogicalNode(INode):
"""
Logical-operator-containing node.
They are internal tree nodes.
"""
def getLogicalOperator():
"""
Returns node's logical operator.
"""
def getNodeList():
"""
Returns the list of subnodes.
"""
class IColumnNode(INode):
"""
Column-name-containing node.
They are internal tree nodes.
Their value applies to any contained ValueNode, except if there is another
ColumnNode between current one and a ValueNode, for which the other
ColumnNode will take precedence.
"""
def getColumnName():
"""
Returns node's column name.
"""
def getSubNode():
"""
Returns node's (only) subnode.
"""
This diff is collapsed.
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class IEntireQuery(Interface):
"""
A EntireQuery represents an entire SQL expression, where a Query
represents only the "WHERE" part of that expression.
A EntireQuery contains:
- a Query instance
- a limit expression
- a group-by expression
- an order-by expression
- a select expression
It internaly uses a ColumnMap instance to resolve tables to use to
generate a "from" expression.
"""
def __init__(query, order_by_list=None, group_by_list=None,
select_dict=None, limit=None, catalog_table_name=None,
extra_column_list=None, from_expression=None,
order_by_override_list=None):
"""
query (Query instance)
The root of the Query tree this query will contain.
order_by_list (list of 1-tuple, 2-tuple or 3-tuple)
The list of columns which will be sorted by SQL.
Tuple values are:
- mandatory: column name
- optionnal: sort order (can be "ASC" or "DESC", "ASC" by default)
- optionnal: type cast (no cast by default, see "CAST" SQL method)
group_by_list (list of string)
The list of columns which will be groupped by value by SQL.
select_dict (dict, key: string, value: string, None)
Given values describe columns to make available in SQL result.
If column is aliased in result set, key is the alias and value is the
column.
Otherwise, key is the column, and value must be None.
limit
See SQLExpression.
catalog_table_name (string)
Name of the table to use as a catalog.
Deprecated parameters.
extra_column_list (list of string)
The list of columns to register to column map. They will not be used
in final rendering, but are hint on which table are supposed to be
used when mapping columns.
from_expression
See SQLExpression.
order_by_override_list (list of string)
If a column is in order_by_list, cannot be mapped to a table column
but is present in this list, it will be passed through to
SQLExpression.
"""
def asSQLExpression(sql_catalog, only_group_columns):
"""
Instantiate a column map, process parameters given at instantiation and
register them to column map.
Register query to column map.
Build column map.
Generate extra SQLExpressions from column map.
Generate SQLExpression instance and return it.
"""
def asSearchTextExpression(sql_catalog):
"""
This is just a passthrough to embeded Query's asSearchTextExpression
method.
This means that only the where expression can be represented as a
SearchText, but not sort, limits, ...
"""
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class IOperator(Interface):
"""
An operator is responsible for rendering a value and a column name as SQL
or Search Text.
This class is designed to be used as a singleton-per-operator.
"""
def __init__(operator):
"""
operator (string)
Operator's text representation. It is used both for SQL and SearchText
rendering.
"""
def asSearchText(value):
"""
Render given value as Search Text
value (see _renderValue)
Value to render as a string for use in a Search Text expression.
"""
def asSQLExpression(column, value_list, only_group_columns):
"""
Construct a SQLExpression instance from given column and value, with
contained glue text.
value_list can be a non-list instance, which must be handled that same
way as a list of one item.
only_group_columns (bool)
If false, the operator can add group columns in the "select_dict" of
returned SLQExpression.
Otherwise, it must not (SQL would be invalid).
"""
def getOperator():
"""
Accessor for operator's text representation.
"""
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class IQuery(Interface):
"""
A Query contains:
- a value
- an operator
- a column
It is the python representation of a predicate, independently of its
rendering (SQL or SearchText).
For SQL rendering to be possible, it is necesary for some data to be
centralized in a data structure known at EntireQuery level (to be able to
generate unique table aliases, for exemple). This is the role of
ColumnMap, and registerColumnMap method on this interface.
This interface also offers various rendering methods, one per rendering
format.
"""
def asSearchTextExpression(sql_catalog, column=None):
"""
Render a query in a user-oriented SearchText.
Returns None if there is this query has no SearchText representation,
but is SearchText-aware.
If column is provided, it must be used instead of local knowledge of
column name. It is used to make queries inside a related key render
correctly.
"""
def asSQLExpression(sql_catalog, column_map, only_group_columns):
"""
Render a query as an SQLExpression instance.
"""
def registerColumnMap(sql_catalog, column_map):
"""
Register a query to given column_map.
"""
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class ISearchKeyCatalog(Interface):
def buildQuery(kw, ignore_empty_string=True, operator='and'):
"""
Build a ComplexQuery from kw values.
kw (dict: string keys, any value)
A query will be emited based on its value. Depending on the type of
the value it is handled differently. Query values will be passed
through to result (key is ignored). For all other types, their key
must be either a known column of a sql_search_tables table, or a
related key name.
- String values will be parsed according to the default SearchKey of
their real column (even for related keys). If parsing was
successful, Queries will be generated from its output.
Otherwise, that value will be taken as such.
- Dictionary values can be composed of the following keys:
'query': Their payload value, considered as empty if not given.
'key': The SearchKey to use for this value, overriding default
column configuration.
(for other possible keys, see SearchKeys)
They will be taken as such.
- All other types will be taken as such, and no "empty" check will be
performed on them.
ignore_empty_string (boolean)
If True, values from kw which are empty will be skipped.
operator (string)
Used to explicit the logical relation between kw entries.
It must be a valid ComplexQuery logical operator ('and', 'or').
"""
def buildSQLQuery(query_table='catalog', REQUEST=None,
ignore_empty_string=1, only_group_columns=False,
limit=None, extra_column_list=None,
**kw):
"""
Construct and return an instance of EntireQuery class from given
parameters by calling buildQuery.
ignore_empty_string (boolean)
See buildQuery.
limit (1-tuple, 2-tuple)
If given, will emit SQL to limit the number of result lines.
group_by_list (list of strings)
If given, will emit SQL to group found lines on given parameter names
(their column if they are column names, corresponding virtual columns
otherwise - as for related keys).
select_dict (dict, key: string, value: string, None)
Given values describe columns to make available in SQL result.
If column is aliased in result set, key is the alias and value is the
column.
Otherwise, key is the column, and value can be None or the same as
key.
select_list (list of strings)
Same as providing select_dict with select_list items as keys, and None
values.
order_by_list (list of 1-, 2-, or 3-tuples of strings)
If given, will emit SQL to sort result lines.
Sort will happen with decreasing precedence in list order.
Given n-tuples can contain those values, always in this order:
- parameter name
- sort order (see SQL documentation of 'ORDER BY')
- type cast (see SQL documentation of 'CAST')
Sort will happen on given parameter name (its column if it's a column
name, corresponding virtual column otherwise - as for related keys).
Extra parameters are passed through to buildQuery.
Backward compatibility parameters:
Those parameters are deprecated and should not be used. They are present
to provide backward compatibility with former ZSQLCatalog version.
REQUEST
Ignored.
extra_column_list (list)
query_table (string, None)
The table to use as catalog table.
If given and None, not catalog table will be used. Use this when you
are using SQLCatalog to generate manualy a part of another query.
That table has a special position in returned query:
- all other tables are joined on this one (when it is required to use
other tables)
- it is expected to have some columns (uid, path)
It is strongly discouraged to use this parameter for any value other
than None.
group_by
Use group_by_list instead.
group_by_expression
Use group_by_list instead.
select_expression
Use select_list or select_dict instead.
sort_on
Use order_by_list instead.
sort_order
Use order_by_list instead.
from_expression
This value will be emited in SQL directly in addition to computed
value.
There is no replacement.
where_expression
This value will be emited in SQL directly in addition to computed
value.
Use Query instances instead.
select_expression_key
This prevents given column from being ignored even if they could not
be mapped.
There is no replacement.
only_group_columns
Replaces former stat__ parameter.
Used to globally disalow use of non-group columns in SQL.
"""
def getSearchKey(column, search_key=None):
"""
Returns the default SearchKey instance for the
requested column. There is one instance per
search_key (incl. virtual keys surch as
source_title) and we try to compute it once
only and cache it.
If search_key is provided, it is used as the
name of the search key class to return.
"""
def getComparisonOperator(operator):
"""
Return a comparison operator matching given string.
String must be a valid SQL comparison operator (=, LIKE, IN, ...).
String case does not matter.
There is one comparison operator instance per possible string value.
"""
# TODO: add support for other operators (logical, ensemblist (?))
def searchResults(REQUEST=None, **kw):
"""
Invokes queryResults with the appropriate
ZSQL Method to return a list of results
"""
def countResults(REQUEST=None, **kw):
"""
Invokes queryResults with the appropriate
ZSQL Method to return a statistics for
a list of results
"""
def queryResults(sql_method, REQUEST=None, src__=0, build_sql_query_method=None, **kw):
"""
Return the result of the given 'sql_method' ZSQL Method after
processing all parameters to build a Query object passed to
that method.
The implementation should do the following.
1- Use **kw parameters to build a Query object
by invoking buildQuery
2- Build a ColumnMap instance by invoking
the buildColumnMap on the Query. Some
optmisation may happen here to try
to build the best possible ColumnMap and
use the best possible indices for joining.
During the ColumnMap build process, the
Search Key associated to each Query node
in the Query tree registers the columns
which are used (ex. to search) or provided
(ex. MATCH value for full text search,
interleave expression or parameter in a
UNION Query)
3- Render the query object as an SQLExpression
instance. This instance contains all necessary
parts to generate:
- where_expression
- sort_expression
- group_by_expression
- select_expression
4- Invoke sql_method
"""
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class ISQLExpression(Interface):
"""
This is a container for chunks of SQL generated by a Query.
Each Query instance generates its own SQLExpression instance.
SQLExpressions can be nested (an SQLExpression can contain 0, 1 or more
SQLExpressions).
Chunks are:
- table_alias_dict
- order_by_list
- group_by_list
- select_dict
- limit
Mutualy exclusive chunks are:
- where expression
- sql_expression_list
Deprecated chunks are:
- from_expression
Providing sql_expression_list with more than one entry makes
where_expression_operator mandatory.
"""
def __init__(query,
table_alias_dict=None,
order_by_list=None,
order_by_dict=None,
group_by_list=None,
where_expression=None,
where_expression_operator=None,
sql_expression_list=None,
select_dict=None,
limit=None,
from_expression=None):
"""
Instantiate an SQLExpression object.
This method does consistency checks on received parameters, so that
failures are detected as early as possible.
Also, it casts most optional parameters into empty lists and empty dicts
to make code using those values simpler.
query (Query)
The Query instance which called this constructor.
table_alias_dict (dict, key: string, value: string)
Table alias dict as returned by ColumnMap.getTableAliasDict() .
order_by_list (list of strings)
List of result ordering, pre-rendered.
order_by_dict (dict, key: string, value: string)
Column rendering replacement specific to order_by.
group_by_list (list of strings)
List of column names on which result line list will be grouped.
where_expression (string)
Text representing a "where" expression of an SQL query.
where_expression_operator ("and", "or", "not", None)
Operator to apply on immediately contained SQLExpressions.
It must be "and" or "or" when there are multiple contained
SQLExpressions, it can be "not" if there is exactly one contained
SQLExpression, and must not be provided if there is no contained
SQLExpression.
sql_expression_list (list of SQLExpression)
List of immediately contained SQLExpressions.
select_dict (dict, key:string, value:string or Null)
Lists all columns to be part of select expression.
Key is column alias.
Value is column name, or Null. If it is Null, the alias will also be
used as column name.
limit (1-tuple, 2-tuple, other)
First item is the number of lines expected, second one if given is the
offset of limited result list within the unlimited result list.
If it is not a tuple it is treated as the value of a 1-tuple
parameter.
from_expression (string)
This parameter is deprecated.
"""
def asSQLExpressionDict():
"""
Returns a dictionnary usable as a **kw for a catalog sql method.
It renders aliases (see getTableAliasDict) as a list of strings and
"from" expression (see getFromExpression) as a list of strings.
See getWhereExpression, getOrderByExpression, getLimitExpression,
getSelectExpression, getGroupByExpression.
"""
##############################################################################
#
# Copyright (c) 2002-2009 Nexedi SA and Contributors. All Rights Reserved.
# Jean-Paul Smets-Solanes <jp@nexedi.com>
# Vincent Pelletier <vincent@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from Interface import Interface
class ISearchKey(Interface):
"""
A SearchKey generates Query instances.
It is responsible for parsing (or not) input value and generating the
appropriate Query structure to match the intended search.
This class is designed as a "singleton per column".
"""
def __init__(column):
"""
column (string)
The column this SearchKey is instanciated for.
"""