See merge request !1497

Reference must always be after the web section path.

This will also forbid using document relative url directly (web_page_module/123).

See merge request !1666

For example, Web Section can contain a Script (Python) to set Access
Rule, but the parent section's objectValues(portal_type='Web Section')
should not include such object.

When there are many simultaneously-pending activities attached to any
processing node family, the
  node>=0
subquery becomes dominant (taking hundreds of time longer than the other
subqueries). As a consequence, this starves processing nodes of activities
and increases the CPU needs of the mariadb process hosting the activity
tables.
So, move this subquery out of the regular codepath, and only run it if no
other subquery found any activity:
- there is no activity preferentially targeting the current node
- there is no activity bound to any of the current node's families
- there is no activity without any node preference at all
Also, simplify the content of that subquery: the effective priority can
only be 3 * priority + 1 when this query is run, and node=0 rows can be
excluded (they should not exist in the current database view).
Also, factorise the logic producing "node=processing_node" and
"node IN node_set" subqueries, for simplicity. In turn, this makes all
family-dependent subqueries use a simple equality test, ensuring a stable
query plan independently from the number of families the current node is
member of.
Also, use "UNION ALL" always, as now:
- all subqueries have stritly distinct result sets
- as per mariadb documentation, "UNION [DISTINCT] applies to all UNIONs
  on the left", so the original comment about where ALL is used was
  incorrect in assuming it was improving the effective query performance
Also, line-split SQL queries as visible in the python source to be more
readable, without effect on the produced SQL.
Also, line-split a few non-trivial python expression to make their
internal structure immediately apparent.

Another effect of this change this change is to reduce activity theft
(activities to be preferentially executed by one node being executed by
another), potentially improving object cache hit-rate and hence decreasing
I/O pressure on the ZODB.

See merge request !1655

`zope_quick_start` was a dtml that was removed in Zope commit c8e4cd5ca
(Simplify control panel objects and actually make them non-persistent.,
2016-08-14), but it is referenced in /index_html page template, which
contains:

    <tal:text tal:replace="structure context/zope_quick_start" />

As a result, rendering / is an error page because of KeyError: 'zope_quick_start'

In SlapOS setup, we are using / in an haproxy health check, so the
health check fail and zope is marked as down and haproxy does not send
requests to backend.

We could have an upgrader constraint to repair this page template, but
because of this, zope is unreachable through haproxy, so it's easier to
ressort to monkey patching.

Zope4 inituser is decoded by decode('utf-8') on Zope startup and it is unicode on python2,
therefore it raises UnicodeDecodeError when searching non ascii charater on catalog.

  Traceback (most recent call last):
  File "<portal_components/test.erp5.testERP5Catalog>", line 4144, in testSearchNonAsciiWithTheInitUser
    person_module.searchFolder(title=person_title)]
  File "./erp5/product/ERP5Type/Core/Folder.py", line 452, in searchFolder
    return self.portal_catalog.searchResults(**kw)
  File "./erp5/product/ERP5Catalog/CatalogTool.py", line 819, in searchResults
    return ZCatalog.searchResults(self, sql_catalog_id=catalog_id, **kw)
  File "./erp5/product/ZSQLCatalog/ZSQLCatalog.py", line 1070, in searchResults
    return catalog.searchResults(REQUEST, **kw)
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2362, in searchResults
    **kw
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2326, in queryResults
    **kw
  File "./erp5/product/ZSQLCatalog/SQLCatalog.py", line 2214, in buildSQLQuery
    only_group_columns,
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 397, in asSQLExpressionDict
    'where_expression': self.getWhereExpression(),
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 298, in getWhereExpression
    result = self.sql_expression_list[0].getWhereExpression()
  File "./erp5/product/ZSQLCatalog/SQLExpression.py", line 303, in getWhereExpression
    result = '(%s)' % (operator.join(x.getWhereExpression() for x in self.sql_expression_list), )
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)

This is necessary on Zope4 because the error page is executed even if
the response already has a locked body - and in this case we don't
want to override the content-type header that might also have been set.

One user of this is Base_redirect from erp5_hal_json_style, when
abort_transaction is True, in that case it sets a body in json, a
content type header of application/json and then raise a redirect so
that the transaction is not commited. This fixes a problem visible on
Zope4 with erp5_bank_reconciliation_renderjs_ui_test:testFunctionalRJSBankReconciliationAction

Even if this is not used because we have locked a status earlier, the
constructor expects an URL with zExceptions 4

This reverts commit bf230f1f.

pushed by mistake

With a fix for the URLs query string being escaped twice.

Also fixes icons not cached in sercice worker for OfficeJS.


See merge request !1661

The logic of the patch accidentally changed in 240a8d26 (
{,Propertied}User: Reduce the overhead from Developer role processing.,
2019-02-14). This used to be a patch to not consider 'Developer' as
a local role in SimpleUser.getRolesInContext , but it was changed to
not consider 'Developer' role at all - also as a global role.

SimpleUser are users from the the root acl_users, users from
erp5/acl_users will typically be PropertiedUsers, because this is a PAS
user folder.

Developer users are exceptional, we should first check if normal user
is allowed before considering Developer role.

zopefoundation/Products.CMFCore commit 882a697 (- correct check for
request compatibility for CookieCrumbler, 2017-05-12)

Restricted python 4 supports iterators

This drops some patches that should no longer be needed with zope 4 to
only keep the publisher timeout changes

this change was made with:

    find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/\\?t=445cf24ebd//g'

This timestamp break officejs offline capabilities.

This reaplies 40f6c8fe ([erp5_xhtml_style] ckeditor: drop hardcoded
timestamp parameter in URL, 2017-10-16) on the updated CKEditor

This was done with:

    find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/"M6K9"/""/g'

M6K9 was the timestamp for this build, that we can see at the top of
ckeditor.js

done with an external method to upload the content of zip file in
skin folder.

    import zipfile
    from io import BytesIO

    def uploadZip(self):
      dest = self.getPortalObject().portal_skins.custom
      with zipfile.ZipFile(
          '/srv/slapgrid/slappart3/srv/project/erp5/ckeditor_4.19.1_4b98d281bc34.zip'
      ) as f:
        for m in f.namelist():
          current_dest = dest
          print(m)
          for part in m.split('/')[:-1]:
            if part not in current_dest.objectIds():
              current_dest.manage_addProduct['OFSP'].manage_addFolder(id=part)
            current_dest = current_dest[part]
          fname = m.split('/')[-1]
          if fname == '_translationstatus.txt':
            fname = 'translationstatus.txt'
          current_dest.manage_addProduct['OFSP'].manage_addFile(
            id=fname, file=BytesIO(f.read(m)))
      return repr(dest)

Using make_query makes sure that the query parameters are properly
encoded and also change selection_index to become an int, because
it keeps the type of parameters.

As a consequence, we had to adjust a few place in the code where
selection_index was tested for truthiness: because "0" as a string is
true, but 0 as an int is not. For that, we changed to test the presence
of selection_name instead of testing selection_index, as they are always
used together.

This fixes a problem that & in URL was encoded twice for listbox
anchor links ( bug_module/1137 )

02011d8e (immediateReindexObject: use super user to reindex script,
2015-12-14) did not apply for inventory, because they were overloading
immediateReindexObject.
Introduce a new level, _immediateReindexObject that will hold the actual
reindexing logic.

Previously this method was using PortalContent.reindexObject which was
monkey patched, to make things less complex and more future proof, move
the monkey patch to a method on base class.

This also drops alternateReindexObject on BalanceTransaction, because it
is already defined in Inventory

This is supposed to fix "Should not load state for ${oid of a skin} when
the connection is closed" sometimes happening with live tests (especially
when the test self.publish and the developer access the site while the
test is suspended on a debugger breakpoint).

The object accessed after the connection is closed was a skin (python
script, sometimes page template or form) that was cached in SKINDATA.
The mechanism to prune entries from the cache uses REQUEST.hold API
which expects that REQUEST.close is called on request, but because
requests were not closed at the end of the request, it happened that
the cache was reused from another connection.

This change to close the requests, like ERP5TypeTestCase is doing in
tearDown (the actual close is done by Testing.ZopeTestCase.connections).

By closing requests at the end of tests, we also have to change so that
at the beginning of the test we initialize the request, by using the same
setSite and setupCurrentSkin that are done in ERP5TypeTestCase.

SKINDATA is an implementation detail that should not be accessed from
this level.

In SkinTemplateItem.install, self._objects contains entries for skin
folders and for all skins. objectValues method calls was called for skin
folders (as expected) and also for all skins, which acquire objectValues
from skin folder and do the work again.

This simplifies this by only running this for skin folders.