Commit 61daff3c authored by Douglas's avatar Douglas

Inventory Pandas: added an initial prototype of a Pandas-based Inventory API

The implementation relies on the Data Array Module. It imports data from the
stocks table through a zSQL Method. Category information is added later in a
column-wise way, so it can be easily done in parallel and query Portal Catalog
once for each category column in the array. This category processing needs to be
done only once, when the array is created, and to new data as it is added.

But there is a catch: each entity that belongs to the movement can have many
categories. So the row can be duplicated for each entity's categories and
searched by equality, or they can be stored as comma-separated values and
searched using a regular expression. Regular expression seems faster for
datasets up to 1M rows.

Some unit tests were also added.

These are the external methods created and their purposes:

- Base_filterInventoryDataFrame is there just to parse keyword arguments and
forward them to Base_getInventoryDataFrame. This is used for the non-programmer
interface of Pandas-based getMovementHistoryList implementation and can be
used as an external method in other scripts too.

- Base_convertResultsToBigArray will convert results of Portal Catalog and ZSQL
Method to a Data Array with a proer transformation of the schema to a
compatible NumPy data type.

- Base_extendBigArray will extend a Data Array with a Portal Catalog query or
ZSQL Method result. Raise errors when the extension data type doesn't match
the source.

- Base_fillPandasInventoryCategoryList will fill category information in a Data
Array which has stock movements information.

-Base_zGetStockByResource is used in a test case as source to create a Data
Array with stock data.
parent 4faae3ab
import pandas as pd
import numpy as np
import re
import transaction
from DateTime import DateTime
from wendelin.bigarray.array_zodb import ZBigArray
class ZBigArrayConverter(object):
'''
ZBigArrayConverter class transforms Portal Catalog or ZSQL Methods results
into a Data Array.
It uses the DtypeIdentifyer class map the structure of the input to a proper
numpy's dtype.
'''
def __init__(self, movements, context):
self.context = context
self.movements = movements
def convert(self, reference=None, overwrite=False, add_category=True):
'''
convert method will do the transformation of the input into a Data Array
itself.
Its parameters are:
- reference: the reference of the Data Array which will be created from the
input;
- overwrite: whether the method should overwrite the Data Array if it
already exists;
- add_category: whether it should add category fields for the movements
entities or not (should only be true if we are dealing with stock
movements);
There is a complication related to how Portal Catalog and ZSQL Methods
order the names of the fields of their results. Both the Results.tuples
and Results.names methods return the fields in a different order. This
is not good when working with an array, so we sort alphabetically the
fields of each row before import it to the Data Array.
'''
my_dtype = DtypeIdentifier(self.movements).identify(add_category=add_category)
size = len(self.movements)
reference = reference or 'WendelinJupyter'
result = self.context.portal_catalog(reference=reference, portal_type='Data Array')
if len(result) == 0:
data_array = self.context.data_array_module.newContent(
portal_type='Data Array',
reference=reference
)
data_array.initArray((size,), my_dtype)
elif not overwrite:
return result[0].getObject()
else:
data_array = result[0]
array = data_array.getArray()
for index in xrange(len(array)):
# We need to order everything related to the data schema here. The Results methods
# `tuples()`, `names` and `data_dictionary` returns the fields in a different order
# and order is very important in the conversion to a ZBigArray. So we build
# an array out of the Results instance with the sorted fields.
ordered_movements = []
for movement in self.movements:
new_movement = [movement[key] for key in sorted(self.movements.names())]
ordered_movements.append(new_movement)
value = [self.filterItem(item, normalize=False) for item in ordered_movements[index]]
value.extend([0] * 10)
array[index:] = tuple(value)
transaction.commit()
return data_array.getObject()
def filterItem(self, item, normalize=False):
'''
Method to proccess values and convert them to an improved representation
that can be used NumPy. Examples are: convertion from DateTime.Datetime
objects to NumPy.datetime64 and None objects to zeroes.
'''
if not item or isinstance(item, type(None)):
return 0
if normalize and isinstance(item, (str, unicode)):
return 0
elif isinstance(item, DateTime):
return np.datetime64(item.ISO8601())
else:
return item
class DtypeIdentifier(object):
'''
DtypeIdentifier class is used to identify the best NumPy.dtype that fits
the input (from the `movements` parameters of the constructor). Then this
dtype will be used later to create a Data Array.
It can include the movements entities' category fields if the `add_category`
parameter of the method `identify` is True.
'''
CATEGORY_LIST = 'resource,node,payment,section,mirror_section,function,project,funding,payment_request,movement'.split(',')
def __init__(self, movements):
self.movements = movements
self.column_data = self.movements.data_dictionary()
self.type_dtype_dict = {
't': 'a',
'l': 'i8',
'd': 'datetime64[s]',
'i': 'i8',
'n': 'f8',
}
def identify(self, add_category=True):
dtypes = self._columns_type_to_dtypes()
names = sorted(self.movements.names())
if add_category:
for _ in range(10):
dtypes.append('a90')
for attribute in self.CATEGORY_LIST:
names.append('%s_category' % attribute)
return np.dtype({
'names': names,
'formats': map(np.dtype, dtypes)
})
def _columns_type_to_dtypes(self):
dtypes = []
for column in sorted(self.column_data):
type_ = self.column_data[column]['type']
size = self.column_data[column]['width']
dtype = self.type_dtype_dict[type_]
if dtype == 'a':
if size > 0:
dtype += str(size)
else:
dtype += '50'
dtypes.append(dtype)
return dtypes
def convertResultsToDataArray(self, results, reference='WendelinJupyter'):
converter = ZBigArrayConverter(results, self)
array = converter.convert(reference=reference)
return array
class ZBigArrayExtender(object):
'''
ZBigArrayExtender class has the purpose of extending an existing ZBigArray
using as input another ZBigArray, a Portal Catalog or ZSQL Method results.
'''
def __init__(self, source, extension):
self.source = source
self.extension = extension
def extend(self):
self.first_part_size = len(self.source)
self.second_part_size = len(self.extension)
self.new_total_size = self.first_part_size + self.second_part_size
self.source.resize((self.new_total_size,))
if isinstance(self.extension, ZBigArray):
self._extend_from_zbigarray()
else:
self._extend_from_results()
return self.source
def _extend_from_zbigarray(self):
if not self.source.dtype == self.extension.dtype:
raise TypeError('Source and extension data types does not match.')
for index, item in enumerate(self.extension):
self.source[index + self.first_part_size:] = item
def _extend_from_results(self):
extension_dtype = DtypeIdentifier(self.extension).identify()
if not self.source.dtype == extension_dtype:
raise TypeError('Source and extension data types does not match.')
for index in xrange(len(self.extension)):
# Basically the same problem here with the order of Results instance fields
# when we convert it to an array.
ordered_movements = []
for movement in self.extension:
new_movement = [movement[key] for key in sorted(self.extension.names())]
ordered_movements.append(new_movement)
self.source[index + self.first_part_size:] = tuple(
[self._filterItem(item, normalize=False) for item in ordered_movements[index]]
)
def _filterItem(self, item, normalize=False):
if not item or isinstance(item, type(None)):
return 0
if normalize and isinstance(item, (str, unicode)):
return 0
elif isinstance(item, DateTime):
return np.datetime64(item.ISO8601())
else:
return item
def extendBigArray(self, source, destination):
return ZBigArrayExtender(
source,
destination
).extend()
class CategoryProcessor(object):
'''
CategoryProcessor class is responsible for filling all the category fields
of a Data Array that holds stock movements information.
For performance reasons, and thanks to NumPy fancy indexing, only one query
to the Portal Catalog is made to get all the categories of a given entity
(check the FIELDS_WITH_CATEGORY constant). So, in total,
`len(FIELDS_WITH_CATEGORY)` queries to Portal Catalog are executed. More
information about the `fillCategories` method can be found in its own
docstring.
It is possible to fill one entity's categories at a time by using the
`fields` parameter of the `fillCategories` method. This allows for category
filling in parallel using activities.
'''
FIELDS_WITH_CATEGORY = 'resource,node,payment,section,mirror_section,function,project,funding,payment_request,source'.split(',')
def __init__(self, array_reference, context):
self.context = context.getPortalObject()
self.array_reference = array_reference
def fillCategoryList(self, fields=None, verbose=False, duplicate_category=False):
'''
FillCategoryList is the proper responsible for filling the category fields
information.
This is the workflow of the method:
1. Iterate over all the entities that can have categories and for each:
1.1. Get all the UIDs of this entity in the whole array and for each:
1.1.1. Get all the categories of this UID using the getCategoryList
method and store in a dictionary with 2 levels: the name entity
where it came from and the UID of this entity
2. Loop over the Data Array and for each of the entities gets the proper
category UID.
It would be good to optimize this method at some point.
'''
fields_with_category = fields or self.FIELDS_WITH_CATEGORY
result = self.context.portal_catalog(portal_type='Data Array', reference=self.array_reference)
array = result[0].getArray()
fields_objects_categories = {}
categories_df = self._getCategoriesDf()
for field in fields_with_category:
fields_objects_categories[field] = {}
if field == 'source':
field_category_name = 'movement_category'
field_name = 'uid'
else:
field_category_name = field+'_category'
field_name = field+'_uid'
if verbose:
print 'Processing %s' % field_name
uids = [str(row[0]) for row in array[:][[field_name]]]
objects = self.context.portal_catalog(uid=uids)
if verbose:
print 'Found %s %s' % (len(objects), field)
for resource in objects:
categories = resource.getCategoryList()
category_uids = []
for category in categories:
try:
category_uids.append(str(categories_df.ix[category]['uid']))
except KeyError:
if verbose:
print 'Category %s not found from %s' % (category, field_category_name)
print '...adding to the DataFrame.'
categories_df.loc[category] = self.context.portal_categories.resolveCategory(category).getUid()
fields_objects_categories[field][int(resource.getUid())] = ','.join(category_uids)
if duplicate_category:
total_duplication = 0
for row in array[0:len(array)]:
if not row['uid']: continue
for field in fields_with_category:
if field == 'source':
field_category_name = 'movement_category'
field_name = 'uid'
else:
field_category_name = field+'_category'
field_name = field+'_uid'
resource_uid = int(row[field_name])
if resource_uid == 0: continue
categories = fields_objects_categories[field][resource_uid]
if duplicate_category:
for category in categories.split(','):
new_size = len(array) + 1
array.resize((new_size,))
array[new_size-1:] = row.copy()
array[new_size-1:][field_category_name] = category
total_duplication += 1
else:
row[field_category_name] = categories
transaction.commit()
if duplicate_category:
print 'Duplication added to the array: %s' % total_duplication
return array
def _getCategoriesDf(self):
'''
_getCategoriesDf creates a Pandas.DataFrame with all categories' UIDs
and indexes it by each category path. It will be later accessed with the
result from the getCategoryList method.
'''
categories = self.context.portal_catalog(portal_type='Category')
categories_path = (category.getPath().split('portal_categories/')[1] for category in categories)
categories_uid = (category.getUid() for category in categories)
return pd.DataFrame(categories_uid, index=categories_path, columns=['uid'])
def fillCategoryList(self, reference, fields=None, verbose=False, duplicate_category=False):
return CategoryProcessor(reference, self).fillCategoryList(
verbose=verbose,
duplicate_category=duplicate_category
)
class InventoryDataFrameQuery(object):
'''
InventoryDataFrameQuery class is responsbiel for queries in a
Pandas.DataFrame created with Data Array filled with stock movements data.
'''
FIELDS_WITH_CATEGORY = 'resource,node,payment,section,mirror_section,function,project,funding,payment_request'.split(',')
def __init__(self, df, context, duplicated_categories=False):
self.df = df
self.context = context
self.duplicated_categories = duplicated_categories
def getMovementHistoryList(self, **kw):
'''
Prototype implementation of the Inventory API `getMovementHistoryList`
using Pandas.DataFrame and Data Array as backend.
Possible arameters are:
* from_date (>=) - only take rows which date is >= from_date
* to_date (<) - only take rows which date is < to_date
* at_date (<=) - only take rows which date is <= at_date
* simulation_state - only take rows where simulation state matches
simulation_state
* input_simulation_state - only take rows with specified simulation_state
and quantity > 0
* output_simulation_state - only take rows with specified simulation_state
and quantity < 0
* only_accountable - Only take into account accountable movements. By
default, only movements for which isAccountable() is
true will be taken into account.
* omit_input - doesn't take into account movement with quantity > 0
* omit_output - doesn't take into account movement with quantity < 0
* omit_asset_increase - doesn't take into account movement with asset_price > 0
* omit_asset_decrease - doesn't take into account movement with asset_price < 0
* resource_uid - only take rows which resource uid matches `resource_uid`
* node_uid - only take rows which node uid matches `node_uid`
* payment_uid - only take rows which payment uid matches `payment_uid`
* section_uid - only take rows which section uid matches `section_uid`
* mirror_section_uid - only take rows which mirror section uid matches `mirror section_uid`
* resource_<category_name>_uid - only take rows where the resource categories at
<category_name> includes resource_<category_name>_uid
* node_<category_name>_uid - only take rows where the node categories at
<category_name> includes node_<category_name>_uid
* payment_<category_name>_uid - only take rows where the payment categories at
<category_name> includes payment_<category_name>_uid
* section_<category_name>_uid - only take rows where the section categories at
<category_name> includes section_<category_name>_uid
* mirror_section_<category_name>_uid - only take rows where the mirror section categories at
<category_name> includes mirror_section_<category_name>_uid
* variation_text - Not implemented yet.
* sub_variation_text - Not implemented yet.
* variation_category - variation or list of possible variations (it is not
a cross-search ; SQL query uses OR)
'''
kw, self.category_kw = self._filterCategoryParameters(**kw)
_, self.raw_filter_dict = self.context.portal_simulation._generateKeywordDict(**kw)
return self._filterDf()
def _filterCategoryParameters(self, **kw):
category_kw = {}
keys_to_delete = []
for key, value in kw.iteritems():
for field in self.FIELDS_WITH_CATEGORY:
regex = re.compile(r'%s_.*_uid$' % field)
if regex.match(key):
field_name = key.split('_')[0] + '_category'
category_kw[field_name] = value
keys_to_delete.append(key)
for key in keys_to_delete:
del kw[key]
return kw, category_kw
def _filterDf(self, duplicated_categories=False):
related_key_dict_filter = self._filterRelatedKeyDictPassthrough()
omit_dict_filter = self._filterOmitDict()
simulation_dict_filter = self._filterSimulationDict()
column_value_dict_filter = self._filterColumnValueDict()
category_filter = self._filterCategories()
return self.df[
column_value_dict_filter &
related_key_dict_filter &
omit_dict_filter &
simulation_dict_filter &
category_filter
]
def _filterRelatedKeyDictPassthrough(self):
accountable = self.raw_filter_dict['related_key_dict_passthrough']['is_accountable']
return self.df['is_accountable'] == int(accountable)
def _filterRelatedKeyDict(self):
pass
def _filterOmitDict(self):
base_node_filter = self.df['node_uid'] != self.df['mirror_node_uid']
base_section_filter = self.df['section_uid'] != self.df['mirror_section_uid']
# check which values pandas will use when one of these fields below are null
# base_null_node_filter = self.df['mirror_node_uid']
# base_null_section_filter = self.df['mirror_section_uid']
# base_payment_filter = self.df['payment_uid']
base_filter = (base_node_filter) & (base_section_filter)
if self.raw_filter_dict['omit_dict']['input'] == 1:
positive_quantity_filter = (self.df['quantity'] >= 0) & (self.df['is_cancellation'] == 0)
negative_quantity_filter = (self.df['quantity'] < 0) & (self.df['is_cancellation'] == 1)
omit_input_output_filter = (positive_quantity_filter) | (negative_quantity_filter)
elif self.raw_filter_dict['omit_dict']['output'] == 1:
positive_quantity_filter = (self.df['quantity'] >= 0) & (self.df['is_cancellation'] == 1)
negative_quantity_filter = (self.df['quantity'] < 0) & (self.df['is_cancellation'] == 0)
omit_input_output_filter = (positive_quantity_filter) | (negative_quantity_filter)
else:
omit_input_output_filter = self._true_array()
if self.raw_filter_dict['omit_dict']['asset_increase'] == 1:
negative_price_filter = (self.df['total_price'] < 0) & (self.df['is_cancellation'] == 0)
positive_price_filter = (self.df['total_price'] >= 0) & (self.df['is_cancellation'] == 1)
omit_increase_decrease_filter = (negative_price_filter) | (positive_price_filter)
elif self.raw_filter_dict['omit_dict']['asset_decrease'] == 1:
negative_price_filter = (self.df['total_price'] < 0) & (self.df['is_cancellation'] == 1)
positive_price_filter = (self.df['total_price'] >= 0) & (self.df['is_cancellation'] == 0)
omit_increase_decrease_filter = (negative_price_filter) | (positive_price_filter)
else:
omit_increase_decrease_filter = self._true_array()
return (base_filter) & (omit_input_output_filter) & (omit_increase_decrease_filter)
def _filterSimulationDict(self):
simulation_states = self.raw_filter_dict['simulation_dict'].get('simulation_state', [])
input_simulation_states = self.raw_filter_dict['simulation_dict'].get('input_simulation_state', [])
output_simulation_states = self.raw_filter_dict['simulation_dict'].get('output_simulation_state', [])
true_array = self._true_array()
# dataframe filter madness starts
if len(simulation_states) == 0:
simulation_state_filter = true_array
else:
simulation_state_filter = self.df['simulation_state'].isin(simulation_states)
if len(input_simulation_states) == 0:
input_simulation_filter = true_array
else:
input_simulation_state_array = (self.df['simulation_state'].isin(input_simulation_states))
input_simulation_quantity_positive_array = (self.df['quantity'] > 0) & (self.df['is_cancellation'] == 0)
input_simulation_quantity_negative_array = (self.df['quantity'] < 0) & (self.df['is_cancellation'] == 1)
input_simulation_filter = (input_simulation_state_array) & (input_simulation_quantity_positive_array | input_simulation_quantity_negative_array)
if len(output_simulation_states) == 0:
output_simulation_filter = true_array
else:
output_simulation_state_array = (self.df['simulation_state'].isin(output_simulation_states))
output_simulation_quantity_positive_array = (self.df['quantity'] > 0) & (self.df['is_cancellation'] == 1)
output_simulation_quantity_negative_array = (self.df['quantity'] < 0) & (self.df['is_cancellation'] == 0)
output_simulation_filter = (output_simulation_state_array) & (output_simulation_quantity_positive_array | output_simulation_quantity_negative_array)
return (simulation_state_filter) | (input_simulation_filter) | (output_simulation_filter)
def _filterColumnValueDict(self):
array_classes = (list, tuple)
columns_values = self.raw_filter_dict['column_value_dict']
final_filter = self._true_array()
for key in columns_values.keys():
if key == 'date':
range_ = columns_values[key]['range']
if range_ == 'minmax':
lower_limit = columns_values[key]['query'][0]
upper_limit = columns_values[key]['query'][1]
date_filter = (self.df['date'] > lower_limit) & (self.df['date'] < upper_limit)
elif range_ == 'min':
lower_limit = columns_values[key]['query'][0]
date_filter = (self.df['date'] > lower_limit)
elif range_ == 'max':
upper_limit = columns_values[key]['query'][1]
date_filter = (self.df['date'] < upper_limit)
final_filter = (date_filter) & (final_filter)
else:
values_to_find = columns_values[key]
values_to_find = values_to_find if isinstance(values_to_find, array_classes) else (values_to_find,)
columns_values_filter = self.df[key].isin(values_to_find)
final_filter = (columns_values_filter) & (final_filter)
return final_filter
def _filterCategories(self):
partial_filter = self._true_array()
for field, value in self.category_kw.iteritems():
if self.duplicated_categories:
partial_filter = (partial_filter) & (self.df[field] == value)
else:
partial_filter = (partial_filter) & (self.df[field].str.contains(r'\b%s\b' % value))
return partial_filter
def _true_array(self):
true_array = np.ones((len(self.df),), dtype=bool)
return true_array
def getInventoryDataFrame(self, data_array_reference=None, duplicated_categories=False, as_csv=False,**kw):
if not data_array_reference:
if duplicated_categories:
data_array_reference = 'WendelinJupyterDuplicated'
else:
data_array_reference = 'WendelinJupyter'
array = self.getPortalObject().portal_catalog(
portal_type='Data Array',
reference=data_array_reference
)[0].getObject().getArray()[:]
df = pd.DataFrame(array)
query_result = InventoryDataFrameQuery(
df,
self,
duplicated_categories=duplicated_categories
).getMovementHistoryList(**kw)
return query_result if not as_csv else query_result.to_csv()
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Extension Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_recorded_property_dict</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>extension.erp5.PandasInventory</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Extension Component</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.patches.WorkflowTool"/>
</pickle>
<pickle>
<tuple>
<none/>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</tuple>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Folder" module="OFS.Folder"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>erp5_wendelin_inventory</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>convertResultsToDataArray</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_convertResultsToBigArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>extendBigArray</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_extendBigArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>fillCategoryList</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_fillPandasInventoryCategoryList</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_filterInventoryDataFrame</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>getInventoryDataFrame</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_getInventoryDataFrame</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
select * from stock
where resource_uid = <dtml-sqlvar resource_uid type=string>
\ No newline at end of file
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="SQL" module="Products.ZSQLMethods.SQL"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_col</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>order_id</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>explanation_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>node_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>section_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>payment_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>function_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>project_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>funding_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>payment_request_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>mirror_section_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>mirror_node_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>resource_uid</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>l</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>7</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>quantity</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>n</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>2</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>is_cancellation</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>i</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>is_accountable</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>i</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>1</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>date</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>d</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>19</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>mirror_date</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>d</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>19</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>total_price</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>n</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>3</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>portal_type</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>t</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>22</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>simulation_state</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>t</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>9</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>variation_text</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>t</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
<dictionary>
<item>
<key> <string>name</string> </key>
<value> <string>sub_variation_text</string> </value>
</item>
<item>
<key> <string>null</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>t</string> </value>
</item>
<item>
<key> <string>width</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</list>
</value>
</item>
<item>
<key> <string>arguments_src</string> </key>
<value> <string>resource_uid</string> </value>
</item>
<item>
<key> <string>connection_id</string> </key>
<value> <string>erp5_sql_connection</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_zGetStockByResource</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>zGetStockByOrder</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
##############################################################################
#
# Copyright (c) 2002-2016 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
##############################################################################
from Products.ERP5Type.tests.ERP5TypeTestCase import ERP5TypeTestCase
import transaction
class PandasInventoryTest(ERP5TypeTestCase):
"""
A test case to ensure the Pandas-based Inventory API works as expected.
"""
def getBusinessTemplateList(self):
"""
Tuple of Business Templates we need to install
"""
return ('erp5_wendelin',)
def afterSetUp(self):
"""
This is ran before anything, used to set the environment
"""
self.supplier = self.createSupplier()
self.client = self.createClient()
self.sale_order = self.createSaleOrder(self.supplier, self.client)
transaction.commit()
self.tic()
self.assertNoPendingMessage()
def createSupplier(self):
organisation_module = self.portal.getDefaultModule(portal_type='Organisation')
organisation = organisation_module.newContent(
title='My Supplier',
)
organisation.validate()
return organisation
def createClient(self):
organisation_module = self.portal.getDefaultModule(portal_type='Organisation')
organisation = organisation_module.newContent(
title='My Client',
)
organisation.validate()
return organisation
def createSaleOrder(self, supplier, client):
sale_order_module = self.portal.getDefaultModule(portal_type='Sale Order')
product_module = self.portal.getDefaultModule(portal_type='Product')
sale_trade_condition_module = self.portal.getDefaultModule(portal_type='Sale Trade Condition')
currency_module = self.portal.getDefaultModule(portal_type='Currency')
currency = currency_module.searchFolder(title='Euro')[0].getObject()
specialise = sale_trade_condition_module.searchFolder(title='General Sale Trade Condition')[0].getObject()
sale_order = sale_order_module.newContent(
source_section_value=supplier,
source_value=supplier,
destination_section_value=client,
destination_value=client,
price_currency=currency.getRelativeUrl(),
start_date='01/01/2015'
)
sale_order.setSpecialiseValue(specialise, portal_type="Sale Trade Condition")
product = product_module.newContent(
title='My product',
quantity_unit='unit/piece'
)
sale_order.newContent(
portal_type='Sale Order Line',
quantity=1,
price=10.0,
resource_value=product
)
self.portal.portal_workflow.doActionFor(
sale_order,
'plan_action',
wf_id='order_workflow'
)
self.portal.portal_workflow.doActionFor(
sale_order,
'confirm_action',
wf_id='order_workflow'
)
return sale_order
def test_01_fillBigArrayTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingFillBigArray'
)
transaction.commit()
self.tic()
data_array = self.portal.portal_catalog(portal_type='Data Array', reference='TestingFillBigArray')[0].getObject()
self.assertEquals(len(data_array.getArray()), 8)
def test_02_extendBigArrayTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingExtendBigArray'
)
transaction.commit()
self.tic()
data_array = self.portal.portal_catalog(portal_type='Data Array', reference='TestingExtendBigArray')[0].getObject()
current_size = len(data_array)
self.portal.portal_skins.erp5_wendelin_inventory.Base_extendBigArray(
data_array.getArray(),
data_array.getArray()
)
self.assertTrue(len(data_array) == 2*current_size)
first_half = data_array.getArray()[0:7]
second_half = data_array.getArray()[8:15]
result = all([x[0] == x[1] for x in zip(first_half, second_half)])
self.assertTrue(result)
def test_03_importCategoryInformationTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingImportCategoryInformation'
)
transaction.commit()
self.tic()
array = self.portal.portal_skins.erp5_wendelin_inventory.Base_fillPandasInventoryCategoryList(
'TestingImportCategoryInformation',
verbose=False,
duplicate_category=False
)
transaction.commit()
self.tic()
resource_category_array = array[:][['resource_category']]
self.assertTrue(all([item != '' for item in resource_category_array]))
def test_04_getMovementHistoryListTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingGetMovementHistoryList'
)
transaction.commit()
self.tic()
result = self.portal.portal_catalog(portal_type='Data Array', reference='TestingGetMovementHistoryList')
self.assertTrue(len(result) != 0)
df = self.portal.portal_skins.erp5_wendelin_inventory.Base_filterInventoryDataFrame(
is_accountable=True,
resource_uid=resource_uid,
data_array_reference='TestingGetMovementHistoryList'
)
self.assertTrue(all(df['is_accountable'] == True))
self.assertTrue(all(df['resource_uid'] == resource_uid))
\ No newline at end of file
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Test Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_recorded_property_dict</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>testPandasInventory</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>test.erp5.testPandasInventory</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Test Component</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.patches.WorkflowTool"/>
</pickle>
<pickle>
<tuple>
<none/>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</tuple>
</pickle>
</record>
</ZopeData>
erp5_wendelin
\ No newline at end of file
extension.erp5.PandasInventory
\ No newline at end of file
portal_components/test.erp5.testPandasInventory
currency_module/
sale_trade_condition_module/1
business_process_module/1
\ No newline at end of file
erp5_wendelin_inventory
\ No newline at end of file
test.erp5.testPandasInventory
\ No newline at end of file
erp5_wendelin_inventory
\ No newline at end of file
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment