See merge request nexedi/erp5!1665

This allows custom movement generators to set ad-hoc properties on
generated movements depending on properties of the corresponding
input movement.

There was already _getUpdatePropertyDict which is similar, but was
sometimes not enough because it can only be use to set extra
properties on all movements.

As a result, when we needed a rule with custom properties on
generated movements, we resorted to overriding getGeneratedMovementList
and duplicating the full logic of the method. Existing cases have
been updated to use the new _updateGeneratedMovementList and in the
case of InventoryAssetPriceAccountingRuleMovementGenerator we no
longer need to duplicate the logic.

Also remove the comment suggesting overriding getGeneratedMovementList
in subclasses, now several "extension" methods exists, so it should
not be needed to override getGeneratedMovementList, every needs
should be coverred by defining _updateGeneratedMovementList or
_getUpdatePropertyDict

This patch does not seem necessary

This SELECT option was added when the UPDATE query was split into a
SELECT + UPDATE query pair, in order to try to avoid deadlocks in mariadb.
  commit 7daaf0a5
  Author: Vincent Pelletier <vincent@nexedi.com>
  Date:   Fri Jun 7 16:53:11 2013 +0200

      Work around poor UPDATE use of index.
That was not completely successful, and another attempt was since made,
which further ipmproved the situation:
  commit e21aa47f601a98f767e2d3e8490b20b7f409933c
  Author: Vincent Pelletier <vincent@nexedi.com>
  Date:   Fri Sep 17 11:34:46 2021 +0900

      CMFActivity.Activity.SQLBase: Reduce the number of deadlocks

This change made the "FOR UPDATE" option obsolete: there should be
virtually no parallelism between the SELECT and UPDATE queries.
By which I mean only 3rd-party accesses (like an admin on a mysql console)
would bypass the SQL lock, causing index-update-level locking issues.

Checking now, I see that removing "FOR UPDATE" on the "node>0" query makes
that query (with LIMIT 30) run about 5 times faster:
With FOR UPDATE:    0.509s
Without FOR UPDATE: 0.090s
This difference is enough to cause significant activity starvation on a
63-processing-nodes cluster where such activities run in about 1s, just
because of the amount of contention on the SQL lock, itself because of the
duration of every execution of this query.

So remove this now-redundant option.

It is expected that this will improve activity processing performance in
general, beyond just the starvation issue which led to this discovery.

See merge request !1497

Reference must always be after the web section path.

This will also forbid using document relative url directly (web_page_module/123).

See merge request !1666

For example, Web Section can contain a Script (Python) to set Access
Rule, but the parent section's objectValues(portal_type='Web Section')
should not include such object.

When there are many simultaneously-pending activities attached to any
processing node family, the
  node>=0
subquery becomes dominant (taking hundreds of time longer than the other
subqueries). As a consequence, this starves processing nodes of activities
and increases the CPU needs of the mariadb process hosting the activity
tables.
So, move this subquery out of the regular codepath, and only run it if no
other subquery found any activity:
- there is no activity preferentially targeting the current node
- there is no activity bound to any of the current node's families
- there is no activity without any node preference at all
Also, simplify the content of that subquery: the effective priority can
only be 3 * priority + 1 when this query is run, and node=0 rows can be
excluded (they should not exist in the current database view).
Also, factorise the logic producing "node=processing_node" and
"node IN node_set" subqueries, for simplicity. In turn, this makes all
family-dependent subqueries use a simple equality test, ensuring a stable
query plan independently from the number of families the current node is
member of.
Also, use "UNION ALL" always, as now:
- all subqueries have stritly distinct result sets
- as per mariadb documentation, "UNION [DISTINCT] applies to all UNIONs
  on the left", so the original comment about where ALL is used was
  incorrect in assuming it was improving the effective query performance
Also, line-split SQL queries as visible in the python source to be more
readable, without effect on the produced SQL.
Also, line-split a few non-trivial python expression to make their
internal structure immediately apparent.

Another effect of this change this change is to reduce activity theft
(activities to be preferentially executed by one node being executed by
another), potentially improving object cache hit-rate and hence decreasing
I/O pressure on the ZODB.

this way they will be expanded in simulation

this will have a side effect that Trade Model Path will also consider
destination carrier, like it does with source_carrier.

Not having destination carrier seems not intentional, during the
early days of "new simulation", source_carrier was added but not
destination_carrier, this was commit d487196a (A first idea of how
BusinessPath will look at the end, 2010-05-22)

Instead of hardcoding categories group in TradeModelPath, introduce
two new groups (as planned in the XXX comments) to list source
and destination arrow base categories.

See merge request !1655

`zope_quick_start` was a dtml that was removed in Zope commit c8e4cd5ca
(Simplify control panel objects and actually make them non-persistent.,
2016-08-14), but it is referenced in /index_html page template, which
contains:

    <tal:text tal:replace="structure context/zope_quick_start" />

As a result, rendering / is an error page because of KeyError: 'zope_quick_start'

In SlapOS setup, we are using / in an haproxy health check, so the
health check fail and zope is marked as down and haproxy does not send
requests to backend.

We could have an upgrader constraint to repair this page template, but
because of this, zope is unreachable through haproxy, so it's easier to
ressort to monkey patching.

Zope4 inituser is decoded by decode('utf-8') on Zope startup and it is unicode on python2,
therefore it raises UnicodeDecodeError when searching non ascii charater on catalog.

  Traceback (most recent call last):
  File "<portal_components/test.erp5.testERP5Catalog>", line 4144, in testSearchNonAsciiWithTheInitUser
    person_module.searchFolder(title=person_title)]
  File "./erp5/product/ERP5Type/Core/Folder.py", line 452, in searchFolder
    return self.portal_catalog.searchResults(**kw)
  File "./erp5/product/ERP5Catalog/CatalogTool.py", line 819, in searchResults
    return ZCatalog.searchResults(self, sql_catalog_id=catalog_id, **kw)
  File "./erp5/product/ZSQLCatalog/ZSQLCatalog.py", line 1070, in searchResults
    return catalog.searchResults(REQUEST, **kw)
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2362, in searchResults
    **kw
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2326, in queryResults
    **kw
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2214, in buildSQLQuery
    only_group_columns,
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 397, in asSQLExpressionDict
    'where_expression': self.getWhereExpression(),
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 298, in getWhereExpression
    result = self.sql_expression_list[0].getWhereExpression()
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 303, in getWhereExpression
    result = '(%s)' % (operator.join(x.getWhereExpression() for x in self.sql_expression_list), )
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)

This is necessary on Zope4 because the error page is executed even if
the response already has a locked body - and in this case we don't
want to override the content-type header that might also have been set.

One user of this is Base_redirect from erp5_hal_json_style, when
abort_transaction is True, in that case it sets a body in json, a
content type header of application/json and then raise a redirect so
that the transaction is not commited. This fixes a problem visible on
Zope4 with erp5_bank_reconciliation_renderjs_ui_test:testFunctionalRJSBankReconciliationAction

Even if this is not used because we have locked a status earlier, the
constructor expects an URL with zExceptions 4

This reverts commit bf230f1f.

pushed by mistake

With a fix for the URLs query string being escaped twice.

Also fixes icons not cached in sercice worker for OfficeJS.


See merge request !1661

The logic of the patch accidentally changed in 240a8d26 (
{,Propertied}User: Reduce the overhead from Developer role processing.,
2019-02-14). This used to be a patch to not consider 'Developer' as
a local role in SimpleUser.getRolesInContext , but it was changed to
not consider 'Developer' role at all - also as a global role.

SimpleUser are users from the the root acl_users, users from
erp5/acl_users will typically be PropertiedUsers, because this is a PAS
user folder.

Developer users are exceptional, we should first check if normal user
is allowed before considering Developer role.

zopefoundation/Products.CMFCore commit 882a697 (- correct check for
request compatibility for CookieCrumbler, 2017-05-12)

Restricted python 4 supports iterators