Skip to content

W
wendelin
Commit 61daff3c authored by Douglas's avatar Douglas
Browse files
Options

Inventory Pandas: added an initial prototype of a Pandas-based Inventory API 

The implementation relies on the Data Array Module. It imports data from the
stocks table through a zSQL Method. Category information is added later in a
column-wise way, so it can be easily done in parallel and query Portal Catalog
once for each category column in the array. This category processing needs to be
done only once, when the array is created, and to new data as it is added.

But there is a catch: each entity that belongs to the movement can have many
categories. So the row can be duplicated for each entity's categories and
searched by equality, or they can be stored as comma-separated values and
searched using a regular expression. Regular expression seems faster for
datasets up to 1M rows.

Some unit tests were also added.

These are the external methods created and their purposes:

- Base_filterInventoryDataFrame is there just to parse keyword arguments and
forward them to Base_getInventoryDataFrame. This is used for the non-programmer
interface of Pandas-based getMovementHistoryList implementation and can be
used as an external method in other scripts too.

- Base_convertResultsToBigArray will convert results of Portal Catalog and ZSQL
Method to a Data Array with a proer transformation of the schema to a
compatible NumPy data type.

- Base_extendBigArray will extend a Data Array with a Portal Catalog query or
ZSQL Method result. Raise errors when the extension data type doesn't match
the source.

- Base_fillPandasInventoryCategoryList will fill category information in a Data
Array which has stock movements information.

-Base_zGetStockByResource is used in a test case as source to create a Data
Array with stock data.
parent 4faae3ab
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1685 additions and 0 deletions
bt5/erp5_wendelin_inventory/ExtensionTemplateItem/portal_components/extension.erp5.PandasInventory.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Extension Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_recorded_property_dict</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>extension.erp5.PandasInventory</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Extension Component</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.patches.WorkflowTool"/>
</pickle>
<pickle>
<tuple>
<none/>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</tuple>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Folder" module="OFS.Folder"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>erp5_wendelin_inventory</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_convertResultsToBigArray.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>convertResultsToDataArray</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_convertResultsToBigArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_extendBigArray.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>extendBigArray</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_extendBigArray</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_fillPandasInventoryCategoryList.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>fillCategoryList</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_fillPandasInventoryCategoryList</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_filterInventoryDataFrame.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_filterInventoryDataFrame</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_getInventoryDataFrame.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>getInventoryDataFrame</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>PandasInventory</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_getInventoryDataFrame</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/SkinTemplateItem/portal_skins/erp5_wendelin_inventory/Base_zGetStockByResource.sql 0 → 100644
View file @ 61daff3c
select * from stock
where resource_uid = <dtml-sqlvar resource_uid type=string>
\ No newline at end of file
bt5/erp5_wendelin_inventory/TestTemplateItem/portal_components/test.erp5.testPandasInventory.py 0 → 100644
View file @ 61daff3c
##############################################################################
#
# Copyright (c) 2002-2016 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
##############################################################################
from Products.ERP5Type.tests.ERP5TypeTestCase import ERP5TypeTestCase
import transaction
class PandasInventoryTest(ERP5TypeTestCase):
"""
A test case to ensure the Pandas-based Inventory API works as expected.
"""
def getBusinessTemplateList(self):
"""
Tuple of Business Templates we need to install
"""
return ('erp5_wendelin',)
def afterSetUp(self):
"""
This is ran before anything, used to set the environment
"""
self.supplier = self.createSupplier()
self.client = self.createClient()
self.sale_order = self.createSaleOrder(self.supplier, self.client)
transaction.commit()
self.tic()
self.assertNoPendingMessage()
def createSupplier(self):
organisation_module = self.portal.getDefaultModule(portal_type='Organisation')
organisation = organisation_module.newContent(
title='My Supplier',
)
organisation.validate()
return organisation
def createClient(self):
organisation_module = self.portal.getDefaultModule(portal_type='Organisation')
organisation = organisation_module.newContent(
title='My Client',
)
organisation.validate()
return organisation
def createSaleOrder(self, supplier, client):
sale_order_module = self.portal.getDefaultModule(portal_type='Sale Order')
product_module = self.portal.getDefaultModule(portal_type='Product')
sale_trade_condition_module = self.portal.getDefaultModule(portal_type='Sale Trade Condition')
currency_module = self.portal.getDefaultModule(portal_type='Currency')
currency = currency_module.searchFolder(title='Euro')[0].getObject()
specialise = sale_trade_condition_module.searchFolder(title='General Sale Trade Condition')[0].getObject()
sale_order = sale_order_module.newContent(
source_section_value=supplier,
source_value=supplier,
destination_section_value=client,
destination_value=client,
price_currency=currency.getRelativeUrl(),
start_date='01/01/2015'
)
sale_order.setSpecialiseValue(specialise, portal_type="Sale Trade Condition")
product = product_module.newContent(
title='My product',
quantity_unit='unit/piece'
)
sale_order.newContent(
portal_type='Sale Order Line',
quantity=1,
price=10.0,
resource_value=product
)
self.portal.portal_workflow.doActionFor(
sale_order,
'plan_action',
wf_id='order_workflow'
)
self.portal.portal_workflow.doActionFor(
sale_order,
'confirm_action',
wf_id='order_workflow'
)
return sale_order
def test_01_fillBigArrayTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingFillBigArray'
)
transaction.commit()
self.tic()
data_array = self.portal.portal_catalog(portal_type='Data Array', reference='TestingFillBigArray')[0].getObject()
self.assertEquals(len(data_array.getArray()), 8)
def test_02_extendBigArrayTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingExtendBigArray'
)
transaction.commit()
self.tic()
data_array = self.portal.portal_catalog(portal_type='Data Array', reference='TestingExtendBigArray')[0].getObject()
current_size = len(data_array)
self.portal.portal_skins.erp5_wendelin_inventory.Base_extendBigArray(
data_array.getArray(),
data_array.getArray()
)
self.assertTrue(len(data_array) == 2*current_size)
first_half = data_array.getArray()[0:7]
second_half = data_array.getArray()[8:15]
result = all([x[0] == x[1] for x in zip(first_half, second_half)])
self.assertTrue(result)
def test_03_importCategoryInformationTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingImportCategoryInformation'
)
transaction.commit()
self.tic()
array = self.portal.portal_skins.erp5_wendelin_inventory.Base_fillPandasInventoryCategoryList(
'TestingImportCategoryInformation',
verbose=False,
duplicate_category=False
)
transaction.commit()
self.tic()
resource_category_array = array[:][['resource_category']]
self.assertTrue(all([item != '' for item in resource_category_array]))
def test_04_getMovementHistoryListTest(self):
resource_uid = self.sale_order['1'].getResourceUid()
data = self.portal.portal_skins.erp5_wendelin_inventory.Base_zGetStockByResource(resource_uid=resource_uid)
self.portal.portal_skins.erp5_wendelin_inventory.Base_convertResultsToBigArray(
data,
reference='TestingGetMovementHistoryList'
)
transaction.commit()
self.tic()
result = self.portal.portal_catalog(portal_type='Data Array', reference='TestingGetMovementHistoryList')
self.assertTrue(len(result) != 0)
df = self.portal.portal_skins.erp5_wendelin_inventory.Base_filterInventoryDataFrame(
is_accountable=True,
resource_uid=resource_uid,
data_array_reference='TestingGetMovementHistoryList'
)
self.assertTrue(all(df['is_accountable'] == True))
self.assertTrue(all(df['resource_uid'] == resource_uid))
\ No newline at end of file
bt5/erp5_wendelin_inventory/TestTemplateItem/portal_components/test.erp5.testPandasInventory.xml 0 → 100644
View file @ 61daff3c
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Test Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_recorded_property_dict</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>testPandasInventory</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>test.erp5.testPandasInventory</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Test Component</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.patches.WorkflowTool"/>
</pickle>
<pickle>
<tuple>
<none/>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</tuple>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_inventory/bt/dependency_list 0 → 100644
View file @ 61daff3c
erp5_wendelin
\ No newline at end of file
bt5/erp5_wendelin_inventory/bt/template_extension_id_list 0 → 100644
View file @ 61daff3c
extension.erp5.PandasInventory
\ No newline at end of file
bt5/erp5_wendelin_inventory/bt/template_keep_path_list 0 → 100644
View file @ 61daff3c
portal_components/test.erp5.testPandasInventory
currency_module/
sale_trade_condition_module/1
business_process_module/1
\ No newline at end of file
bt5/erp5_wendelin_inventory/bt/template_skin_id_list 0 → 100644
View file @ 61daff3c
erp5_wendelin_inventory
\ No newline at end of file
bt5/erp5_wendelin_inventory/bt/template_test_id_list 0 → 100644
View file @ 61daff3c
test.erp5.testPandasInventory
\ No newline at end of file
bt5/erp5_wendelin_inventory/bt/title 0 → 100644
View file @ 61daff3c
erp5_wendelin_inventory
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7