Skip to content

erp5
erp5
Commit 316e1147 authored by Douglas's avatar Douglas Committed by Ivan Tyagov
Browse files
Options

ERP5 Jupyter kernel improvements and integration of PivotTableJs 

Please, review this: @Tyagov, @kirr and @tatuya. 

## Kernel improvements: 

* Automatically render last-returning objects as HTML through "processors" 
  - These so called "processors" are classes responsible for rendering certain objects as HTML depending on their type 
  - The user can add his own customized processors, either by editing the JupyterCompile extension itself or using the variable `_processor_list` in the code cells

## ERP5 Jupyter kernel integration with [PivotTableJs](https://github.com/nicolaskruchten/pivottable)

* Implemented as an external method that receives a Pandas.DataFrame as parameter, along with the url of the ERP5 instance (is there a way to detect this instead of hardcoding all the time?)
* Works through an IPython.lib.display.IFrame object, that is added to the result of the code cell. Inside this IFrame the pivot table will be rendered with data from the DataFrame
* The IFrame is stored/hosted in the (volatile, for now) memcached sever of the instance, using his own HTML representation as key and accessed through a Script (Python) object

## Notes

There's more detailed information about the implementation in the commit messages and a little bit more in comments inside the JupyterCompile extension, where the kernel lives.

The demo web page included in the bt5 will only work if the Wendelin software release is installed, as it depends partially on things from the ERP5 repository and part from things on the following merge request to Wendelin: nexedi/wendelin!10 . Should I move it there or keep it here?

~~I'm still refactoring to completely remove those global variables from the code. Will add the commit here ASAP.~~ 

Global variables completely removed!


/reviewed-on nexedi/erp5!63
parents b2003f09 9cfc4f66
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 704 additions and 177 deletions
bt5/erp5_data_notebook/ExtensionTemplateItem/portal_components/extension.erp5.JupyterCompile.xml
View file @ 316e1147
......@@ -46,14 +46,14 @@
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple>
<string>W: 58, 2: Using the global statement (global-statement)</string>
<string>W:104, 8: Use of exec (exec-used)</string>
<string>W:129, 10: Use of exec (exec-used)</string>
<string>W:146, 10: Use of exec (exec-used)</string>
<string>W:208, 2: Unused variable \'etype\' (unused-variable)</string>
<string>W:208, 16: Unused variable \'tb\' (unused-variable)</string>
<string>W:269, 4: Using the global statement (global-statement)</string>
<string>W:369, 2: Using the global statement (global-statement)</string>
<string>W: 85, 2: Redefining name \'traceback\' from outer scope (line 9) (redefined-outer-name)</string>
<string>W:197, 8: Use of exec (exec-used)</string>
<string>W:252, 8: Use of exec (exec-used)</string>
<string>W:264, 8: Use of exec (exec-used)</string>
<string>W:327, 10: Unused variable \'mime_type\' (unused-variable)</string>
<string>W:462, 2: Redefining name \'IFrame\' from outer scope (line 17) (redefined-outer-name)</string>
<string>W: 9, 0: Unused import traceback (unused-import)</string>
<string>W: 13, 0: Unused import transaction (unused-import)</string>
</tuple>
</value>
</item>
......
bt5/erp5_data_notebook/PathTemplateItem/portal_caches/erp5_pivottable_frame_cache.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Cache Factory" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_count</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>_mt_index</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>_tree</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAQ=</string> </persistent>
</value>
</item>
<item>
<key> <string>cache_duration</string> </key>
<value> <int>36000</int> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>erp5_pivottable_frame_cache</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Cache Factory</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>PivotTableJs Frames Cache</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="Length" module="BTrees.Length"/>
</pickle>
<pickle> <int>0</int> </pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="OOBTree" module="BTrees.OOBTree"/>
</pickle>
<pickle>
<none/>
</pickle>
</record>
<record id="4" aka="AAAAAAAAAAQ=">
<pickle>
<global name="OOBTree" module="BTrees.OOBTree"/>
</pickle>
<pickle>
<none/>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/PathTemplateItem/portal_caches/erp5_pivottable_frame_cache/3.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Distributed Ram Cache" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>specialise/portal_memcached/default_memcached_plugin</string>
</tuple>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>3</string> </value>
</item>
<item>
<key> <string>int_index</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Distributed Ram Cache</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Distributed Volatile RAM based cache plugin</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_displayHTML.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>Base_displayHTML</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_displayHTML</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_displayPivotTableFrame.py 0 → 100644
View file @ 316e1147
cache_factory = context.getPortalObject().portal_caches.erp5_pivottable_frame_cache
return cache_factory.get(key)
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_displayPivotTableFrame.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>key</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_displayPivotTableFrame</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>displayPivotTableFrame</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_erp5PivotTableUI.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>erp5PivotTableUI</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_erp5PivotTableUI</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/ERP5Site_getExamplePivotTableJsMovementHistoryList.xml 0 → 100644
View file @ 316e1147
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ZopePageTemplate" module="Products.PageTemplates.ZopePageTemplate"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>content_type</string> </key>
<value> <string>text/html</string> </value>
</item>
<item>
<key> <string>expand</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_getExamplePivotTableJsMovementHistoryList</string> </value>
</item>
<item>
<key> <string>output_encoding</string> </key>
<value> <string>utf-8</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <unicode></unicode> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/ERP5Site_getExamplePivotTableJsMovementHistoryList.zpt 0 → 100644
View file @ 316e1147
<html>
<head>
<script type="text/javascript" tal:attributes="src python: context.portal_url() + '/jquery/core/jquery.min.js'" src=""></script>
<script type="text/javascript" tal:attributes="src python: context.portal_url() + '/jquery/ui/js/jquery-ui.min.js'" src=""></script>
<script type="text/javascript" src="http://evanplaice.github.io/jquery-csv/src/jquery.csv.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/d3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/c3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/export_renderers.min.js"></script>
<script type="text/javascript">
$(document).ready(function () {
$('#filter_button').on('click', function (){
var data = $('form').serializeArray();
var url = $('form').data('portal-url');
$('#pivottablejs').html('Loading...');
$.ajax({
type: "POST",
url: url + '/Base_filterInventoryDataFrame?as_csv=True',
data: data,
success: function (response) {
var input = $.csv.toArrays(response);
$('#pivottablejs').pivotUI(input, {
renderers: $.extend(
$.pivotUtilities.renderers,
$.pivotUtilities.c3_renderers,
$.pivotUtilities.d3_renderers,
$.pivotUtilities.export_renderers
),
hiddenAttributes: [""],
rows: 'Sequence',
cols: 'Data'
});
},
error: function (response) {
$('#pivottablejs').html('Error while requesting data from server.');
}
})
});
});
</script>
</head>
<body>
<h1>Integration between Pandas-based Inventory API and PivotTableJS</h1>
<p><b>NOTE: for this protoype the code will use the Big Array object with title "Wendelin + Jupyter"</b></p>
<form tal:attributes="data-portal-url context/portal_url">
<fieldset>
<legend>Is accountable?</legend>
<input type="radio" name="is_accountable" value="1" checked> Yes
<input type="radio" name="is_accountable" value="0"> No
</fieldset>
<fieldset>
<legend>Omit</legend>
<input type="checkbox" name="omit_input"> Omit Input
<input type="checkbox" name="omit_output"> Omit Output
<input type="checkbox" name="omit_asset_increase"> Omit Asset Increase
<input type="checkbox" name="omit_asset_decrease"> Omit Asset Decrease
</fieldset>
<fieldset>
<legend>Simulation State</legend>
<p>Simulation State: <input name="simulation_state"></p>
<p>Input Simulation State: <input name="input_simulation_state"></p>
<p>Output Simulation State: <input name="output_simulation_state"></p>
</fieldset>
<fieldset>
<legend>Dates</legend>
<p>From date (yyyy-mm-dd): <input name="from_date"></p>
<p>To date (yyyy-mm-dd): <input name="to_date"></p>
</fieldset>
<button type="button" id="filter_button">Filter!</button>
</form>
<div id="pivottablejs">
</div>
</body>
</html>
\ No newline at end of file
bt5/erp5_data_notebook/TestTemplateItem/portal_components/test.erp5.testExecuteJupyter.py
View file @ 316e1147
......@@ -73,7 +73,9 @@ class TestExecuteJupyter(ERP5TypeTestCase):
def testJupyterCompileErrorRaise(self):
"""
Test if JupyterCompile portal_component raises error on the server side.
Test if JupyterCompile portal_component correctly catches exceptions as
expected by the Jupyter frontend as also automatically abort the current
transaction.
Take the case in which one line in a statement is valid and another is not.
"""
portal = self.getPortalObject()
......@@ -365,19 +367,20 @@ import sys
self.assertEquals(json.loads(result)['code_result'].rstrip(), 'imghdr')
self.assertEquals(json.loads(result)['mime_type'].rstrip(), 'text/plain')
def testBaseDisplayImageERP5Image(self):
def testERP5ImageProcessor(self):
"""
Test the fucntioning of Base_displayImage external method of erp5_data_notebook
BT5 for ERP5 image object as parameter and change
Test the fucntioning of the ERP5ImageProcessor and the custom system
display hook too.
"""
self.image_module = self.portal.getDefaultModule('Image')
self.assertTrue(self.image_module is not None)
# Create a new ERP5 image object
reference = 'testBase_displayImageReference'
reference = 'testBase_displayImageReference5'
data_template = '<img src="data:application/unknown;base64,%s" /><br />'
data = 'qwertyuiopasdfghjklzxcvbnm<somerandomcharacterstosaveasimagedata>'
self.image_module.newContent(
portal_type='Image',
id='testBase_displayImageID',
id='testBase_displayImageID5',
reference=reference,
data=data,
filename='test.png'
......@@ -387,7 +390,7 @@ import sys
# Call Base_displayImage from inside of Base_runJupyter
jupyter_code = """
image = context.portal_catalog.getResultValue(portal_type='Image',reference='%s')
context.Base_displayImage(image_object=image)
context.Base_renderAsHtml(image)
"""%reference
local_variable_dict = {'imports' : {}, 'variables' : {}}
......@@ -396,7 +399,7 @@ context.Base_displayImage(image_object=image)
old_local_variable_dict=local_variable_dict
)
self.assertEquals(result['result_string'].rstrip(), base64.b64encode(data))
self.assertEquals(result['result_string'].rstrip(), data_template % base64.b64encode(data))
# Mime_type shouldn't be image/png just because of filename, instead it is
# dependent on file and file data
self.assertNotEqual(result['mime_type'], 'image/png')
......@@ -434,4 +437,34 @@ context.Base_displayImage(image_object=image)
reference=reference,
python_expression=jupyter_code2
)
self.assertEquals(json.loads(result)['code_result'].rstrip(), 'sys')
\ No newline at end of file
self.assertEquals(json.loads(result)['code_result'].rstrip(), 'sys')
def testPivotTableJsIntegration(self):
'''
This test ensures the PivotTableJs user interface is correctly integrated
into our Jupyter kernel.
'''
portal = self.portal
self.login('dev_user')
jupyter_code = '''
class DataFrameMock(object):
def to_csv(self):
return "column1, column2; 1, 2;"
my_df = DataFrameMock()
iframe = context.Base_erp5PivotTableUI(my_df)
context.Base_renderAsHtml(iframe)
'''
reference = 'Test.Notebook.PivotTableJsIntegration %s' % time.time()
notebook = self._newNotebook(reference=reference)
result = portal.Base_executeJupyter(
reference=reference,
python_expression=jupyter_code
)
json_result = json.loads(result)
# The big hash in this string was previous calculated using the expect hash
# of the pivot table page's html.
pivottable_frame_display_path = 'Base_displayPivotTableFrame?key=853524757258b19805d13beb8c6bd284a7af4a974a96a3e5a4847885df069a74d3c8c1843f2bcc4d4bb3c7089194b57c90c14fe8dd0c776d84ce0868e19ac411'
self.assertTrue(pivottable_frame_display_path in json_result['code_result'])
bt5/erp5_data_notebook/TestTemplateItem/portal_components/test.erp5.testExecuteJupyter.xml
View file @ 316e1147
......@@ -43,7 +43,9 @@
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
<tuple>
<string>W:457, 4: Unused variable \'notebook\' (unused-variable)</string>
</tuple>
</value>
</item>
<item>
......
bt5/erp5_data_notebook/bt/description
View file @ 316e1147
Interaction between Jupyter(IPython Notebook) and ERP5.
!WARNING!
This business template is unsafe to install on a public server as one of the extensions uses eval and allows remote code execution. Proper security should be taken into account.
This business template is unsafe to install on a public server as one of the extensions uses eval and allows remote code execution. Proper security should be taken into account.
This template includes a highly exprimental integration with PivotTableJs which doesn't follow ERP5 Javascript standards and will be refactored to use JIO and RenderJS.
!WARNING!
\ No newline at end of file
bt5/erp5_data_notebook/bt/template_path_list 0 → 100644
View file @ 316e1147
portal_caches/erp5_pivottable_frame_cache
portal_caches/erp5_pivottable_frame_cache/**
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7