Documentation says this step is done in one transaction, so implement
this as documented.

Detecting developer mistake and automagically running this step in smaller
transactions if more than 100 documents seems unnecessary. At least, it
contredicts documentation.

now that we use alarm sense method, another `notify` call is activated

This should fix some race conditions which made post-upgrade steps not
executed sometimes.

* each steps use an defaultActivateParameterDict so that activity they
  may create inherits the same tag, so that next step also runs after
  this). This makes it possible for example to use post-upgrade step on
  documents newly installed in upgrade step.
* use tag / after tag between upgrader steps instead of checking if
  previous alarm had errors. We also now allow force running only one
  step regardless of the previous step completion state, there was no
  strong reason to prevernt this.
* do not use two different active sense and fix method for alarms, just
  use an active sense method understanding `fixit` parameter

Not changed by Ni or Sven in b1f5a740

When a Property Sheet with a ScriptConstraint (whose Document is a ZODB
Component) was added to Memcached Tool: Document Component was partially
loaded with no Property Sheet yet (and thus without the accessors needed
to import ScriptConstraint ZODB Component) and ScriptConstraint Document
was considered not available.

Changes from Ni Yan, and Sven Franck

Changes from Ni Yan

Otherwise this is interpreted as text/html after sending html emails

knowledge_pad/worklist gadget: support displaying a worklist that does not respect the (%(count)s) convention

* Worklist must include (%(count)s) to include the number of document in that worklist
* The role of users who have to take care of the documents must be defined on the worklist

There's no magic in this patch series: it is known that HBTreeFolder2 has
limitations about the ids that can be set without conflict, and this can't be
fixed without causing compatibility issues with existing data.

The patches contain:
- some optimization
- bug fixes
- detection of id conflicts before causing data loss

This will also allow us to use a newer version of ZODB. Recent BTrees failed
on the following line of `_setOb`:

        if len(id_list) == 1 and not htree.has_key(None):

(None is not valid key for comparison since
 ZODB commit bb5aac21277f43333d6450064dc6670c8c280e40)

The long story about id conflicts is that a HBTreeFolder2 can't store 2 objects
<A> and <A>-<B> where <A> does not contain '-', and that's the rule followed by
_getOb/_setOb/_delOb. However:
- Conflicts are detected by testing the type of the value, which means
  HBTreeFolder2 can't store values of the same type as the one it uses
  internally (i.e. OOBTree).
- For performance reasons, _htree_iteritems and getTreeIdList use a stricter
  rule: they assume there can't be 2 objects <A> and <A>-<B>, regardless of the
  presence of a separator in <A>. Maybe this rule should be enforced in _setOb.

/reviewed-on nexedi/erp5!112

It was an unefficient adaptation of BTreeFolder2 code.

These methods have probably never been used.

getTreeIdList is still required by Folder_reindexAll.
getTreeIdList is not immediate anymore but the reindexing of a folder should
anyway be reimplemented to not depend on this method: only one tree is indexed
at a time, which is unefficient when they're small.

_getOb and similar methods are reimplemented in a faster and safer way.
It now checks it is only used to return leafs.

Similarly, _delOb now refuses to delete trees at the root.

__getattr__ wrongly returned wrapped results (__of__).

Since commit 055d0a69
("HBTreeFolder2: make object{Ids,Values,Items} really lazy"),
_fixCount() does nothing because objectIds() was optimized in a way that
len(self.objectIds()) already returns self._count()

Callers up the stack should not mutate returned value (and all callers
found in this repository code follow this rule).
Saves a list copy and an extra call on each call, and a few lines of code.

work partially done by Vincent Pelletier (z_get_table_schema)

@Tyagov, please, check the demo notebook added.



/reviewed-on nexedi/erp5!145

erp5_web_renderjs_ui: add a common gadget to render multi&relation field & no need to show "Create:X" string

/reviewed-on nexedi/erp5!140

/reviewed-on nexedi/erp5!139

@Tyagov review, please. I'm creating a test suite now and will post about the test results as soon as they are available. 

- An environment object was implemented to help us deal with the multiprocess
architecture of ERP5 and objects that cannot be easily stored in the ZODB.
It stores definition of functions, classes and variables as string. The implementation
uses a dumb Environment class to allow users to make `define` and `undefine` calls,
which are captured and processed by an AST transformer before code execution.

- Along with the environment object, an automatic "import fixer" was created. It does
not allow users to import modules as they normally would, because this may cause
collateral effects on other users' code. A good example is the plot settings in the
matplotlib module. It will fix normal imports, make them use the environment object
mentione earlier automatically and warn the user about it.

Some bugs were fixed with this new implementation: 

- https://nexedi.erp5.net/bug_module/20160318-7098DD, which reports an inconsistency
on portal catalog queries between Jupyter and Python (Script) objects. Probably an
issue with user context storage in ActiveProcess

- https://nexedi.erp5.net/bug_module/20160330-13AA193, which reports an error related
to acquisition when trying to plot images, which happened in other situations, although
this is not officially reported in Nexedi's ERP5. This probably also was happening because
of old user context storage.


/reviewed-on nexedi/erp5!131

Variables are investigated, recursively in case of container objects (like lists, for example),
to detect if they can be stored in the ZODB.
In this investigation persistent objects are identified by being an instance of the object
class and implementing a `__getstate__` method that raises no exception. If the variable is
not a Persistent object then we try to pickle and load it.

While developing the pickleable object identification a complication was found. It seems that
the code cannot capture cPickle.PicklingError in the usual way, `except cPickle.PicklingError`.
It's consequence of some weirdness with regards to pickle/cPickle modules exceptions classes and
more about it can be read at http://bugs.python.org/issue1457119. So, the workaround for this complication
was to catch all exceptions and check the exception class name as string.

The whole check for zodb persistence was moved into an utility function for the sake of readability
and code maintenance.

The Base_executeJupyter script object was transformed into an extension to be able to properly handle
transaction errors and render them correctly inside Jupyter.

Avoid getting all related values when we jump to a module.

Getter of content translations were setting an empty translation dict. So
there was ZODB pollution by doing useless changes.