Commit c32ff174 authored by Douglas's avatar Douglas

Jupyter: Added experimental integration between pivottablejs and Pandas.DataFrame

pivottablejs is a very useful pivot table implementation in Javascript that
alllows the user to create his own tables and charts. And also they had examples
of integration with Pandas.DataFrame objects and Jupyter. So this is highly
based on that.

**ATTENTION**: this is an experimental integration and does not follow the ERP5
Javascript standards. It will be refactored in the future to use RenderJS and
JIO.

The integration generates an HTML page template which starts the pivot table and
have a placeholder for the data, that will be later replaced with a Data Frame
data as CSV. After this replacement the page is stored in the memcached server
and then served from there, through a Script (Python) object, inside an HTML
iframe. The iframe is necessary because a lot of Javascript libraries that are
not included in the Jupyter web page are loaded.

A web page with id "PivotTableJs_getMovementHistoryList" was created to demo
how pivottablejs can be integrated within ERP5, either using AJAX or not.

In the process of this integration a simple external method to render
iPython's display classes (Images, Video, Youtube, IFrame, etc) was created. It
will be refactored and polished along with the kernel itself in the future.
parent f235c6b0
...@@ -14,6 +14,110 @@ mime_type = 'text/plain' ...@@ -14,6 +14,110 @@ mime_type = 'text/plain'
status = u'ok' status = u'ok'
ename, evalue, tb_list = None, None, None ename, evalue, tb_list = None, None, None
def Base_executeJupyter(self, python_expression=None, reference=None, title=None, request_reference=False, **kw):
  • Can you please add a docstring explaining the meaning of parameters here ?

  • Sure, but just clarifying: this entire file was not added by me, 99% of it was already there. I don't know why this entire function is marked as added, as it was written by Ayush and merged some time ago.

  • Oh right, I also felt it curious that this function was added.

    I see that in erp5 master branch, Base_executeJupyter exists in portal_skins ( from https://lab.nexedi.com/nexedi/erp5/blob/master/bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_executeJupyter.py added since bbae0fe5 ), but never existed in https://lab.nexedi.com/nexedi/erp5/commits/master/bt5/erp5_data_notebook/ExtensionTemplateItem/portal_components/extension.erp5.JupyterCompile.py (at least on master branch).

    I don't know how this function happeared in your working copy, but I don't think you should commit it. In my understanding this function is not used.

    You may want to ask @tiwariayush to be sure.

  • @jerome @Camata : Yeah, looks like there has been some mistake here. This function was never inside the JupyterCompile extension. It's always been a part of portal skin from where it calls the external functions defined in the extension. Keeping it here is not required at all.

  • Thanks for confirming, @tiwariayush. I already had it removed in a new commit.

Please register or sign in to reply
context = self
portal = context.getPortalObject()
# Check permissions for current user and display message to non-authorized user
if not portal.Base_checkPermission('portal_components', 'Manage Portal'):
return "You are not authorized to access the script"
import json
# Convert the request_reference argument string to their respeced boolean values
request_reference = {'True': True, 'False': False}.get(request_reference, False)
  • FYI, zope has a special convention of passing types parameters to scripts http://docs.zope.org/zope2/zope2book/ScriptingZope.html#passing-parameters-to-scripts

    This implies the caller will use ?request_reference:boolean=True

  • But we do not have control of the script caller here, which is the Jupyter frontend. So we cannot force this script to be called with ?request_reference:boolean=True unless we modify Jupyter's code.

  • I guess the code calling this endpoint and passing request_reference is also our code, but this is very minor and OK as it is

Please register or sign in to reply
# Return python dictionary with title and reference of all notebooks
# for request_reference=True
if request_reference:
data_notebook_list = portal.portal_catalog(portal_type='Data Notebook')
  • I don't know the datamodel well, but are we sure that there will not be "too many" data notebooks here ?

  • Probably yes, but I also don't know what this does exactly. It looks a bit not ideal indeed though.

Please register or sign in to reply
notebook_detail_list = [{'reference': obj.getReference(), 'title': obj.getTitle()} for obj in data_notebook_list]
  • FYI, one minor optimisation is to use:

    [{'reference': brain.reference, 'title': brain.title} for brain in 
       portal.portal_catalog(select_dict={'reference': None, 'title':None}, portal_type='Data Notebook')

    calling obj.getReference() when obj is a catalog brain will get the object from ZODB, brain.reference just use the reference from the index. This only works for indexed properties, so use this trick only if performance became a problem.

Please register or sign in to reply
return notebook_detail_list
if not reference:
message = "Please set or use reference for the notebook you want to use"
return message
# Take python_expression as '' for empty code from jupyter frontend
if not python_expression:
python_expression = ''
# Get Data Notebook with the specific reference
data_notebook = portal.portal_catalog.getResultValue(portal_type='Data Notebook',
reference=reference)
# Create new Data Notebook if reference doesn't match with any from existing ones
if not data_notebook:
notebook_module = portal.getDefaultModule(portal_type='Data Notebook')
data_notebook = notebook_module.DataNotebookModule_addDataNotebook(
title=title,
reference=reference,
batch_mode=True
)
# Add new Data Notebook Line to the Data Notebook
data_notebook_line = data_notebook.DataNotebook_addDataNotebookLine(
notebook_code=python_expression,
batch_mode=True
)
# Get active_process associated with data_notebook object
process_id = data_notebook.getProcess()
active_process = portal.portal_activities[process_id]
# Add a result object to Active Process object
result_list = active_process.getResultList()
# Get local variables saves in Active Result, local varibales are saved as
# persistent mapping object
old_local_variable_dict = result_list[0].summary
if not old_local_variable_dict:
old_local_variable_dict = context.Base_addLocalVariableDict()
# Pass all to code Base_runJupyter external function which would execute the code
# and returns a dict of result
final_result = Base_compileJupyterCode(self, python_expression, old_local_variable_dict)
code_result = final_result['result_string']
new_local_variable_dict = final_result['local_variable_dict']
ename = final_result['ename']
evalue = final_result['evalue']
traceback = final_result['traceback']
status = final_result['status']
mime_type = final_result['mime_type']
# Call to function to update persistent mapping object with new local variables
# and save the variables in the Active Result pertaining to the current Data Notebook
new_dict = context.Base_updateLocalVariableDict(new_local_variable_dict)
result_list[0].edit(summary=new_dict)
result = {
u'code_result': code_result,
u'ename': ename,
u'evalue': evalue,
u'traceback': traceback,
u'status': status,
u'mime_type': mime_type
}
# Catch exception while seriaizing the result to be passed to jupyter frontend
# and in case of error put code_result as None and status as 'error' which would
# be shown by Jupyter frontend
try:
serialized_result = json.dumps(result)
except UnicodeDecodeError:
result = {
u'code_result': None,
u'ename': u'UnicodeDecodeError',
u'evalue': None,
u'traceback': None,
u'status': u'error',
u'mime_type': mime_type
}
serialized_result = json.dumps(result)
data_notebook_line.edit(notebook_code_result=code_result, mime_type=mime_type)
return serialized_result
def Base_compileJupyterCode(self, jupyter_code, old_local_variable_dict): def Base_compileJupyterCode(self, jupyter_code, old_local_variable_dict):
""" """
Function to execute jupyter code and update the local_varibale dictionary. Function to execute jupyter code and update the local_varibale dictionary.
...@@ -122,6 +226,7 @@ def Base_compileJupyterCode(self, jupyter_code, old_local_variable_dict): ...@@ -122,6 +226,7 @@ def Base_compileJupyterCode(self, jupyter_code, old_local_variable_dict):
for node in to_run_interactive: for node in to_run_interactive:
mod = ast.Interactive([node]) mod = ast.Interactive([node])
code = compile(mod, '<string>', "single") code = compile(mod, '<string>', "single")
context = self
exec(code, g, g) exec(code, g, g)
# Letting the code fail in case of error while executing the python script/code # Letting the code fail in case of error while executing the python script/code
...@@ -189,6 +294,22 @@ def UpdateLocalVariableDict(self, existing_dict): ...@@ -189,6 +294,22 @@ def UpdateLocalVariableDict(self, existing_dict):
for key, val in existing_dict['imports'].iteritems(): for key, val in existing_dict['imports'].iteritems():
new_dict['imports'][key] = val new_dict['imports'][key] = val
return new_dict return new_dict
def Base_displayHTML(self, node):
"""
External function to identify Jupyter display classes and render them as
HTML. There are many classes from IPython.core.display or IPython.lib.display
that we can use to display media, like audios, videos, images and generic
HTML/CSS/Javascript. All of them hold their HTML representation in the
`_repr_html_` method.
"""
if getattr(node, '_repr_html_'):
global mime_type
mime_type = 'text/html'
html = node._repr_html_()
print html
return
def Base_displayImage(self, image_object=None): def Base_displayImage(self, image_object=None):
""" """
...@@ -324,3 +445,82 @@ def getError(self, previous=1): ...@@ -324,3 +445,82 @@ def getError(self, previous=1):
tb_list = [l+'\n' for l in error['tb_text'].split('\n')] tb_list = [l+'\n' for l in error['tb_text'].split('\n')]
return None return None
def storeIFrame(self, html, key):
memcached_tool = self.getPortalObject().portal_memcached
memcached_dict = memcached_tool.getMemcachedDict(key_prefix='pivottablejs', plugin_path='portal_memcached/default_memcached_plugin')
memcached_dict[key] = html
return True
def erp5PivotTableUI(self, df, erp5_url):
from IPython.display import IFrame
template = """
<!DOCTYPE html>
<html>
<head>
<title>PivotTable.js</title>
<!-- external libs from cdnjs -->
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery-csv/0.71/jquery.csv-0.71.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/d3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/c3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/export_renderers.min.js"></script>
<style>
body {font-family: Verdana;}
.node {
border: solid 1px white;
font: 10px sans-serif;
line-height: 12px;
overflow: hidden;
position: absolute;
text-indent: 2px;
}
.c3-line, .c3-focused {stroke-width: 3px !important;}
.c3-bar {stroke: white !important; stroke-width: 1;}
.c3 text { font-size: 12px; color: grey;}
.tick line {stroke: white;}
.c3-axis path {stroke: grey;}
.c3-circle { opacity: 1 !important; }
</style>
</head>
<body>
<script type="text/javascript">
$(function(){
if(window.location != window.parent.location)
$("<a>", {target:"_blank", href:""})
.text("[pop out]").prependTo($("body"));
$("#output").pivotUI(
$.csv.toArrays($("#output").text()),
{
renderers: $.extend(
$.pivotUtilities.renderers,
$.pivotUtilities.c3_renderers,
$.pivotUtilities.d3_renderers,
$.pivotUtilities.export_renderers
),
hiddenAttributes: [""]
}
).show();
});
</script>
<div id="output" style="display: none;">%s</div>
</body>
</html>
"""
html_string = template % df.to_csv()
from hashlib import sha512
key = sha512(html_string).hexdigest()
storeIFrame(self, html_string, key)
url = "%s/Base_displayPivotTableFrame?key=%s" % (erp5_url, key)
iframe = IFrame(src=url, width='100%', height='500')
return Base_displayHTML(self, iframe)
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Web Page" module="erp5.portal_type"/>
  • I would suggest using a OFS.File or a page template in portal_skins instead of a web page. Web page are made for data modified by the end users, not for the source code.

Please register or sign in to reply
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_Access_contents_information_Permission</string> </key>
<value>
<tuple>
<string>Anonymous</string>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Add_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Change_local_roles_Permission</string> </key>
<value>
<tuple>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_Modify_portal_content_Permission</string> </key>
<value>
<tuple>
<string>Assignee</string>
<string>Assignor</string>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>_View_Permission</string> </key>
<value>
<tuple>
<string>Anonymous</string>
<string>Assignee</string>
<string>Assignor</string>
<string>Associate</string>
<string>Auditor</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>classification/collaborative/team</string>
</tuple>
</value>
</item>
<item>
<key> <string>content_md5</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>content_type</string> </key>
<value> <string>text/html</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>PivotTableJs_getMovementHistoryList</string> </value>
</item>
<item>
<key> <string>language</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Web Page</string> </value>
</item>
<item>
<key> <string>short_title</string> </key>
<value> <string>getMovementHistoryList</string> </value>
</item>
<item>
<key> <string>text_content</string> </key>
<value> <string encoding="cdata"><![CDATA[
<html>\n
<head>\n
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/core/jquery.min.js"></script>\n
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/ui/js/jquery-ui.min.js"></script>\n
  • localhost 🤓

  • Oh gosh, I forgot these links. Thanks!

  • Well, this is the webpage, so the file shouldn't be here at all. I'll check again if it was correctly removed in favor of the page template version.

Please register or sign in to reply
<script type="text/javascript" src="http://evanplaice.github.io/jquery-csv/src/jquery.csv.js"></script>\n
\n
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.css">\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.js"></script>\n
\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.js"></script>\n
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.css">\n
\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/d3_renderers.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/c3_renderers.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/export_renderers.min.js"></script>\n
\n
<script type="text/javascript">\n
$(document).ready(function () {\n
$(\'#filter_button\').on(\'click\', function (){\n
var data = $(\'form\').serializeArray();\n
$(\'#pivottablejs\').html(\'Loading...\');\n
\n
$.ajax({\n
type: "POST",\n
url: "http://localhost:2200/erp5/portal_skins/erp5_inventory_pandas/filterDataFrame?as_csv=True",\n
  • also, if you make it a page template, you will be able to use portal_url instead of hardcoding localhost here.

  • @jerome awesome suggestion, thanks. I hated that I had to hard code this. Will take a look into this as soon as possible.

  • @jerome replaced this web page object with a page template object in the commit b4873388

    Edited by Douglas
Please register or sign in to reply
data: data,\n
success: function (response) {\n
var input = $.csv.toArrays(response);\n
$(\'#pivottablejs\').pivotUI(input, {\n
renderers: $.extend(\n
$.pivotUtilities.renderers, \n
$.pivotUtilities.c3_renderers, \n
$.pivotUtilities.d3_renderers,\n
$.pivotUtilities.export_renderers\n
),\n
hiddenAttributes: [""],\n
rows: \'Sequence\',\n
cols: \'Data\'\n
});\n
},\n
error: function (response) {\n
$(\'#pivottablejs\').html(\'Error while requesting data from server.\');\n
}\n
})\n
});\n
});\n
</script>\n
</head>\n
<body>\n
<h1>Integration between Pandas-based Inventory API and PivotTableJS</h1>\n
<p><b>NOTE: for this protoype the code will use the Big Array object with title "Wendelin + Jupyter"</b></p>\n
\n
<form>\n
<fieldset>\n
<legend>Is accountable?</legend>\n
<input type="radio" name="is_accountable" value="1" checked> Yes\n
<input type="radio" name="is_accountable" value="0"> No\n
</fieldset>\n
\n
<fieldset>\n
<legend>Omit</legend>\n
<input type="checkbox" name="omit_input"> Omit Input\n
<input type="checkbox" name="omit_output"> Omit Output\n
<input type="checkbox" name="omit_asset_increase"> Omit Asset Increase\n
<input type="checkbox" name="omit_asset_decrease"> Omit Asset Decrease\n
</fieldset>\n
\n
<fieldset>\n
<legend>Simulation State</legend>\n
<p>Simulation State: <input name="simulation_state"></p>\n
<p>Input Simulation State: <input name="input_simulation_state"></p>\n
<p>Output Simulation State: <input name="output_simulation_state"></p>\n
</fieldset>\n
\n
<fieldset>\n
<legend>Dates</legend>\n
<p>From date (yyyy-mm-dd): <input name="from_date"></p>\n
<p>To date (yyyy-mm-dd): <input name="to_date"></p>\n
</fieldset>\n
\n
<button type="button" id="filter_button">Filter!</button>\n
</form>\n
\n
<div id="pivottablejs">\n
</div>\n
</body>\n
</html>
]]></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Web Interface for getMovementHistoryList</string> </value>
</item>
<item>
<key> <string>version</string> </key>
<value>
<none/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>Base_displayHTML</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_displayHTML</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_body</string> </key>
<value> <string>memcached_tool = context.getPortalObject().portal_memcached\n
memcached_dict = memcached_tool.getMemcachedDict(key_prefix=\'pivottablejs\', plugin_path=\'portal_memcached/default_memcached_plugin\')\n
return memcached_dict[key]\n
</string> </value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>key</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_displayPivotTableFrame</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>displayPivotTableFrame</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>erp5PivotTableUI</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_erp5PivotTableUI</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
web_page_module/PivotTableJs_getMovementHistoryList
\ No newline at end of file
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment