Skip to content

E
erp5
Commit a0d21451 authored by Douglas's avatar Douglas Committed by Ivan Tyagov
Browse files
Options

erp5_data_notebook: environment object implementation and refactoring to ERP5 kernel 

@Tyagov review, please. I'm creating a test suite now and will post about the test results as soon as they are available. 

- An environment object was implemented to help us deal with the multiprocess
architecture of ERP5 and objects that cannot be easily stored in the ZODB.
It stores definition of functions, classes and variables as string. The implementation
uses a dumb Environment class to allow users to make `define` and `undefine` calls,
which are captured and processed by an AST transformer before code execution.

- Along with the environment object, an automatic "import fixer" was created. It does
not allow users to import modules as they normally would, because this may cause
collateral effects on other users' code. A good example is the plot settings in the
matplotlib module. It will fix normal imports, make them use the environment object
mentione earlier automatically and warn the user about it.

Some bugs were fixed with this new implementation: 

- https://nexedi.erp5.net/bug_module/20160318-7098DD, which reports an inconsistency
on portal catalog queries between Jupyter and Python (Script) objects. Probably an
issue with user context storage in ActiveProcess

- https://nexedi.erp5.net/bug_module/20160330-13AA193, which reports an error related
to acquisition when trying to plot images, which happened in other situations, although
this is not officially reported in Nexedi's ERP5. This probably also was happening because
of old user context storage.


/reviewed-on !131
parents fb52f4a5 23e06437
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 792 additions and 325 deletions
bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/local_varibale_property.xml → bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/notebook_context_property.xml
View file @ a0d21451
......@@ -10,7 +10,7 @@
<key> <string>categories</string> </key>
<value>
<tuple>
<string>elementary_type/string</string>
<string>elementary_type/object</string>
</tuple>
</value>
</item>
......@@ -22,7 +22,7 @@
</item>
<item>
<key> <string>id</string> </key>
<value> <string>local_varibale_property</string> </value>
<value> <string>notebook_context_property</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_addLocalVariableDict.xml → bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_createNotebookContext.xml
View file @ a0d21451
......@@ -8,7 +8,7 @@
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>AddNewLocalVariableDict</string> </value>
<value> <string>createNotebookContext</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
......@@ -16,7 +16,7 @@
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_addLocalVariableDict</string> </value>
<value> <string>Base_createNotebookContext</string> </value>
</item>
<item>
<key> <string>title</string> </key>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_executeJupyter.py deleted 100644 → 0
View file @ fb52f4a5
"""
Python script to create Data Notebook or update existing Data Notebooks
identifying notebook by reference from user.
Expected behaviour from this script:-
1. Return unauthorized message for non-developer user.
2. Create new 'Data Notebook' for new reference.
3. Add new 'Data Notebook Line'to the existing Data Notebook on basis of reference.
4. Return python dictionary containing list of all notebooks for 'request_reference=True'
"""
portal = context.getPortalObject()
# Check permissions for current user and display message to non-authorized user
if not portal.Base_checkPermission('portal_components', 'Manage Portal'):
return "You are not authorized to access the script"
import json
# Convert the request_reference argument string to their respeced boolean values
request_reference = {'True': True, 'False': False}.get(request_reference, False)
# Return python dictionary with title and reference of all notebooks
# for request_reference=True
if request_reference:
data_notebook_list = portal.portal_catalog(portal_type='Data Notebook')
notebook_detail_list = [{'reference': obj.getReference(), 'title': obj.getTitle()} for obj in data_notebook_list]
return notebook_detail_list
if not reference:
message = "Please set or use reference for the notebook you want to use"
return message
# Take python_expression as '' for empty code from jupyter frontend
if not python_expression:
python_expression = ''
# Get Data Notebook with the specific reference
data_notebook = portal.portal_catalog.getResultValue(portal_type='Data Notebook',
reference=reference)
# Create new Data Notebook if reference doesn't match with any from existing ones
if not data_notebook:
notebook_module = portal.getDefaultModule(portal_type='Data Notebook')
data_notebook = notebook_module.DataNotebookModule_addDataNotebook(
title=title,
reference=reference,
batch_mode=True
)
# Add new Data Notebook Line to the Data Notebook
data_notebook_line = data_notebook.DataNotebook_addDataNotebookLine(
notebook_code=python_expression,
batch_mode=True
)
# Get active_process associated with data_notebook object
process_id = data_notebook.getProcess()
active_process = portal.portal_activities[process_id]
# Add a result object to Active Process object
result_list = active_process.getResultList()
# Get local variables saves in Active Result, local varibales are saved as
# persistent mapping object
old_local_variable_dict = result_list[0].summary
if not old_local_variable_dict:
old_local_variable_dict = context.Base_addLocalVariableDict()
# Pass all to code Base_runJupyter external function which would execute the code
# and returns a dict of result
final_result = context.Base_runJupyter(python_expression, old_local_variable_dict)
code_result = final_result['result_string']
new_local_variable_dict = final_result['local_variable_dict']
ename = final_result['ename']
evalue = final_result['evalue']
traceback = final_result['traceback']
status = final_result['status']
mime_type = final_result['mime_type']
# Call to function to update persistent mapping object with new local variables
# and save the variables in the Active Result pertaining to the current Data Notebook
new_dict = context.Base_updateLocalVariableDict(new_local_variable_dict)
result_list[0].edit(summary=new_dict)
result = {
u'code_result': code_result,
u'ename': ename,
u'evalue': evalue,
u'traceback': traceback,
u'status': status,
u'mime_type': mime_type
}
# Catch exception while seriaizing the result to be passed to jupyter frontend
# and in case of error put code_result as None and status as 'error' which would
# be shown by Jupyter frontend
try:
serialized_result = json.dumps(result)
except UnicodeDecodeError:
result = {
u'code_result': None,
u'ename': u'UnicodeDecodeError',
u'evalue': None,
u'traceback': None,
u'status': u'error',
u'mime_type': mime_type
}
serialized_result = json.dumps(result)
data_notebook_line.edit(notebook_code_result=code_result, mime_type=mime_type)
return serialized_result
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_executeJupyter.xml
View file @ a0d21451
......@@ -2,71 +2,26 @@
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_Access_contents_information_Permission</string> </key>
<value>
<tuple>
<string>Authenticated</string>
<string>Author</string>
<string>Manager</string>
<string>Owner</string>
</tuple>
</value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
<key> <string>_function</string> </key>
<value> <string>Base_executeJupyter</string> </value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>python_expression=None, reference=None, title=None, request_reference=False, **kw</string> </value>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_executeJupyter</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
......
bt5/erp5_data_notebook/PropertySheetTemplateItem/portal_property_sheets/JupyterNotebook/process_property.xml → bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_getErrorMessageForException.xml
View file @ a0d21451
......@@ -2,31 +2,25 @@
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Standard Property" module="erp5.portal_type"/>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>elementary_type/string</string>
</tuple>
</value>
<key> <string>_function</string> </key>
<value> <string>getErrorMessageForException</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>process_property</string> </value>
<value> <string>Base_getErrorMessageForException</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Standard Property</string> </value>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_runJupyter.xml
View file @ a0d21451
......@@ -8,7 +8,7 @@
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>Base_compileJupyterCode</string> </value>
<value> <string>Base_runJupyterCode</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/DataNotebookModule_addDataNotebook.py
View file @ a0d21451
"""
Python script to add a new notebook to Data Notebook module.
This script also concerns for assigning an Active Process for each data notebook
created.
This script also concerns for assigning an empty notebook context for each data
notebook created.
"""
from Products.CMFActivity.ActiveResult import ActiveResult
# Comment out person in case addition of person required to Data Notebook object
#person = context.ERP5Site_getAuthenticatedMemberPersonValue()
# Create new ActiveProcess object and getting its id
active_process = context.portal_activities.newActiveProcess()
active_process_id = active_process.getId()
# Creating new dictionary via external method to save results in ZODB
new_dict = context.Base_addLocalVariableDict()
# Add new ActiveResult object and add it to the activeprocess concerned with ...
# Data Notebook in concern
result = ActiveResult(summary=new_dict)
active_process.activateResult(result)
# Creating new context via external method to save results in ZODB
notebook_context = context.Base_createNotebookContext()
# Create new notebook
notebook = context.newContent(
title=title,
reference=reference,
process=active_process_id,
notebook_context=notebook_context,
portal_type='Data Notebook'
)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7