See merge request nexedi/erp5!1919

   * const is handled as enum with a single value automatically
   * patch ensureArraySize to not modify the user input, let the user safely slice by himself.
   * Reorganise patches so it is grouped by prototype

Improvements compared to the previous implementation:
- avoid looking up the user document again, when the PAS plugin already did
  that job
- make it possible to call a single script when multiple sources of groups
  are based on the same documents, avoiding iterating unnecessarily on
  those same documents multiple times
- avoid repeating the same membership value (ex: when a user has multiple
  assignments with a common membership subset)
- avoid resolving the same relation more than once
- do not go from document value to relative URL only to go from relative
  URL back to document value at the next step
- move security group id extraction to unrestricted python, as the security
  overhead was taking a large amount of time
In a security setup with 8 scripts (all Assignment-based), 6 base
categories, and 4 Assignments (all valid), this implementation is 10 times
faster at producing the same group id set as the previous one.

See merge request nexedi/erp5!1918

   This prevents loose data and wrong edits.

   XXX update description on a squash to explain better.

This is no longer needed now, maybe since Zope4 or maybe since a patch
in slapos. This option does not exist on python3 and using pylint on
python3 complains about these disable, so we have to remove them on
python2 as well.

See merge request nexedi/erp5!1912

This makes consistent method resolution order with python3 and reveal a
problem that relation field widgets render_pdf was not the expected one
on python3, solve this by defining explicitly the method.

roundDebitCredit does not raise in case of big difference since
696bc0d2 (2009-02-06 yo * Stop raising an exception, even if the
credit is not equal to the debit in a built delivery, in the post
processing script, because the builder may not blindly adopt prevision
when updating existing lines, thus it can be unbalanced., 2009-02-06)

This assertRaises syntax does nothing on python2 and was an error on
python2. Update the test to test the actual behavior

When trying to use `MemcachedTool` without the `python-memcached` package,
a dummy implementation is supposed to allow installation while giving a
warning when called.

This dummy implementation was broken because it was calling a wrong class,
named `_MemcachedTool`, which did not exist. Instead, `_MemcacheTool` did
exist and was used for the real implementation.

This commit renames `_MemcacheTool` to `_MemcachedTool` to allow dummy
implementation to work.

Fixes an logged error while indexing:

    2024-03-07 23:44:21.997 WARNING CMFActivity Message dropped (no object found at path ('', 'erp5_portal_ec7d28d9ada99a2cad998125d59fc158', 'image_module', 'gadget_officejs_appstore_app_text_editor_icon_text_editor_png'))
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart22/t/eje/soft/de456d85f70e1b1a086396326977c83c/parts/erp5/product/CMFActivity/ActivityTool.py", line 262, in getObject
        obj = self._getObject(activity_tool)
      File "/srv/slapgrid/slappart22/t/eje/soft/de456d85f70e1b1a086396326977c83c/parts/erp5/product/CMFActivity/ActivityTool.py", line 256, in _getObject
        obj = obj[id]
      File "/srv/slapgrid/slappart22/t/eje/soft/de456d85f70e1b1a086396326977c83c/eggs/Zope-4.8.9+slapospatched002-py2.7.egg/OFS/ObjectManager.py", line 843, in __getitem__
        raise KeyError(key)
    KeyError: 'gadget_officejs_appstore_app_text_editor_icon_text_editor_png'

   Workflows now are viable from ERP5 UI, so redirect to /view instead

With python2, iterating on a dictionary or a set always produces the same result,
although this is not a documented behavior. On python3 this is not the case,
because the hashing algorithm is random by default, which can also be set using [`PYTHONHASHSEED`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED). On SlapOS, this is done with slapos!1535 

This fixes the parts where ERP5 code depends on python2 order, mostly tests, but also places
where we iterate on a dictionary or set. Most of the time, the fix has been to sort so that
the order is deterministic regardless of the hash algorithm randomization, but sometimes we
had to extend a bit the configuration where the order was really important. We did this after
discovering the problematic areas by running tests multiple times with different hash randomization
seeds. It's not impossible that changing from "default python2 order" to "sorted" reveals some
more problems in custom configurations, but this would mean that the configuration must be 
adjusted to use explicit order instead of being lucky with the default python2 order.

The main pattern was the use of `edit` method which edits properties in an order that is a bit
constrained with the `edit_order` mechanism, because some properties depend on other properties,
so it's important to set them in order. This extends a bit the `edit_order` mechanism to specify
more properties that were edited in the right order with `PYTHONHASHSEED=0` by chance.

This also extends delivery builders to edit properties in order defined in the equivalence tester,
most equivalence tester were already properly configured, except the `start_date` and `stop_date`
from delivery level movement groups. That probably only matters for some specific test assertions,
but in practice this was visible in a lot of failing tests.

Some visible changes are that:
 - workflows are now sorted alphabetically on history tab
 - properties are now sorted alphabetically on the diff view of history tab
 - business templates are installed in the order of dependencies and in alphabetic order when they
  are not constrained.

See merge request !1882

  Remove forbidden properties when retrieve the properties from the schema.

      - template and options isn't  part of json schema spec, so it isn't possible to use this feature globally.
      - template also could be used to call callbacks, so despite we block unsafe-eval, it still better remove it.
      - both were removed because it can lead to parameter injection, where by saving the form w/o editing anything, it changes the parameters, it adds non-visible values, which can up to some extend be a security risk.

   Update the description to display the "default" value as a hint, if it was provided into the schema.

constructs like `portal_catalog(title=d.keys())` were allowed on
python2, we can allow them on python3 as well.

See discussion in https://www.erp5.com/group_section/forum/Worklists-with-security-uid-columns-wa7gzLN6NG

See merge request nexedi/erp5!1712

With some combinations of worklists using two additional security_uid
columns coming from local role groups, some documents might be excluded
from worklists, without the fix, when running with a random
PYTHONHASHSEED, the test can fail with:

    FAIL: test_worklist_exclusionlist_collision (erp5.component.test.erp5_version.testERP5CatalogSecurityUidOptimization.TestSecurityUidOptimizationWorklist)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "<portal_components/test.erp5.testERP5CatalogSecurityUidOptimization>", line 370, in test_worklist_exclusionlist_collision
        'security_uid_or_alternate_security_uid_draft': 1,
      File "<portal_components/test.erp5.testERP5CatalogSecurityUidOptimization>", line 213, in assertWorklistCount
        expected_count_by_worklist_id,
    AssertionError: {'security_uid_or_alternate_security_uid_draft': 1} != {'collision_worklist': 1, 'security_uid_or_alternate_security_uid_draft': 1}
    - {'security_uid_or_alternate_security_uid_draft': 1}
    + {'collision_worklist': 1, 'security_uid_or_alternate_security_uid_draft': 1}
    ?  +++++++++++++++++++++++++

What happens in sumCatalogResultByWorklist is something like this:

    (Pdb) pp criterion_dict
    {'alternate_security_uid': <ExclusionList [11, 14]>,
     'other_security_uid': frozenset([13L]),
     'portal_type': frozenset(['Person']),
     'validation_state': frozenset(['draft'])}

    (Pdb) pp catalog_result.dictionaries()
    [{'alternate_security_uid': 12,
      'count': Decimal('1'),
      'other_security_uid': 13,
      'portal_type': 'Person',
      'validation_state': 'draft'}]

Depending on the worklists grouped together in grouped_worklist_dict and
the order this dict is iterated, we may reach a situation where
criterion_id_list contains 'alternate_security_uid', because for another
worklist it was not an ExclusionList but a list to be applied, in this
case, this if condition is false:

        for criterion_id in criterion_id_list:
          criterion_value_set = criterion_dict[criterion_id]
          if result_line[criterion_id] not in criterion_value_set:
            is_candidate = False

and the row is not counted in collision_worklist worklist.
Co-authored-by: Jérome Perrin <jerome@nexedi.com>

These are printed like a normal list, so it's a bit hard to figure out
if they are supposed to be excluded when debugging

This refactors security_uid_innodb_catalog test to have the config in a
test business template and include a workflow with some worklists.

This also depends on erp5_worklist_sql, to test worklists when a
combination of erp5_security_uid_innodb_catalog and erp5_worklist_sql
are enabled, which a common situation.

Otherwise we get an error when just adding a worklist

See merge request nexedi/erp5!1900

   This business template introduces a way to adjust mariadb and data.fs after restore from backup. In the occasion that the backups are unsync (which is normal) it helps to push both on sync instead of fully reindex the site.