erp5_data_notebook: Don't save ZBigArray in data notebook. It may be too big...

erp5_data_notebook: Don't save ZBigArray in data notebook. It may be too big that zope process may crash.

bt5/erp5_data_notebook/ExtensionTemplateItem/portal_components/extension.erp5.JupyterCompile.py

@@ -30,7 +30,8 @@ display_data_wrapper_lock = threading.Lock()
 
 # Well known unserializable types
 from Record import Record
-well_known_unserializable_type_tuple = (ModuleType, Record,)
+from wendelin.bigarray.array_zodb import ZBigArray
+well_known_unserializable_type_tuple = (ModuleType, Record, ZBigArray)
 
 def Base_executeJupyter(self, python_expression=None, reference=None, \
                         title=None, request_reference=False, **kw):

bt5/erp5_data_notebook/ExtensionTemplateItem/portal_components/extension.erp5.JupyterCompile.xml

@@ -46,13 +46,13 @@
             <key> <string>text_content_warning_message</string> </key>
             <value>
               <tuple>
-                <string>W:395, 10: Use of exec (exec-used)</string>
-                <string>W:441, 10: Use of exec (exec-used)</string>
-                <string>W:454, 10: Use of exec (exec-used)</string>
-                <string>W:543,  4: No exception type(s) specified (bare-except)</string>
-                <string>W:551,  6: No exception type(s) specified (bare-except)</string>
-                <string>W:863,  6: Use of exec (exec-used)</string>
-                <string>W:1098,  2: Redefining name \'IFrame\' from outer scope (line 4) (redefined-outer-name)</string>
+                <string>W:396, 10: Use of exec (exec-used)</string>
+                <string>W:442, 10: Use of exec (exec-used)</string>
+                <string>W:455, 10: Use of exec (exec-used)</string>
+                <string>W:544,  4: No exception type(s) specified (bare-except)</string>
+                <string>W:552,  6: No exception type(s) specified (bare-except)</string>
+                <string>W:864,  6: Use of exec (exec-used)</string>
+                <string>W:1099,  2: Redefining name \'IFrame\' from outer scope (line 4) (redefined-outer-name)</string>
               </tuple>
             </value>
         </item>

[Kazuhiko Shiozaki]

This change requires wendelin.core and causes failures in erp5_data_notebook:testExecuteJupyter test :

(test output)
  File "/SR/eggs/Zope2-2.13.26-py2.7.egg/App/Extensions.py", line 175, in getObject % module)
NotFound: The specified module, 'JupyterCompile', couldn't be found.

(zLOG.log)
  File "<portal_components/extension.erp5.JupyterCompile>", line 33, in <module>                
    from wendelin.bigarray.array_zodb import ZBigArray                                          
ImportError: erp5.component.extension.JupyterCompile: cannot load Component JupyterCompile (No module named wendelin.bigarray.array_zodb)

Which do you prefer ?

• use try...except ImportError here and add ZBigArray only if import succeeds
• put (at least) wendelin.core in ERP5 SR's eggs.

/cc @yusei, @jm, @kirr, @Tyagov

[Yusei Tahara]

Thanks. I will add try...except. This commit itself is a temporary fix.

[Kirill Smelkov]

@yusei, first of all I appologize for the delay with replying to this. The notification was not received on my side it seems.

However I'd like to ask: what is the reason ZBigArray needs to be not included into a notebook? ZBigArray is just a plain ZODB object with persistent references to other objects.

Probably the real bug here is that CanSerialize wrongly fallbacks to serializing without persistent references (and then it tries to create huge pickle for whole array content) or something like this.

If we cannot store arrays in the notebook it significantly reduces Jupyter usefullness, imho, at least in scientific context.

Could you please re-look into this issue?

Thanks beforehand,
Kirill

/cc @Tyagov, @klaus (hopefully notification will be delivered)

[Yusei Tahara]

This is a temporary fix so that I am not hit by a known bug. I need to review a function that serializes an object in JupyterCompile, I will do it once I have time, probably next month. The mechanism of storing variables in notebook must be reviewed and be fixed.

[Kirill Smelkov]

@yusei thanks for feedback. I once again did not received your response and got it only because I was curious and went here to check.

/cc @alain.takoudjou @rafael

[Kirill Smelkov]

@yusei, btw it would be better to mark in the code itself the fix is temporary, e.g. with XXX or FIXME so that the current state is clear. Thanks.

[Yusei Tahara]

@kirr OK. I will put a mark next time. Thanks.

[Kirill Smelkov]

@yusei I went ahead and added the FIXME myself in d8924d48.

/cc @luke

[Yusei Tahara]

Thank you.