See merge request !2001

See merge request !2003

In 1b555dbf (py2/py3: Make Products code compatible with both python2
and python3 (nexedi/erp5!1751)., 2024-07-16) we changed the code in a
way that on python2 it would first node text to a python2 str and only
later convert it to unicode, but setting text to a str during the first
step can fail with:

    ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

This change to set directly unicode on py2.

and move the implementation of SimulationMovement.getSimulationState
here. This makes it possible for rules to generate simulation movements
that are not necessary planned.

Applied rules must have a rule as specialise

`sender@customer.com <sender@customer.com>` used in the test is not a
valid email address. We have updated to python3.9.20 which comes with a
fix for CVE-2023-27043 and no longer allow this kind of broken addresses.

Replace the address with a similar valid address,
`"sender@customer.com" <sender@customer.com>`, that was probably the
original intention of this test.

On python3, the type of selected columns depend on the data type from
mariadb side, VARCHAR will be str, BINARY/BLOB will be bytes, etc

These SQL method select path that is first evaluated from a variable
that is NULL and in that case, mariadb seems to select LONGBLOB as data
type:

    MariaDB [test]> set @defined_as_null=null; drop table if exists tmp; create table tmp as (select @defined_as_null); show create table tmp;
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | Table | Create Table                                                                                                                       |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+
    | tmp   | CREATE TABLE `tmp` (
      `@defined_as_null` longblob DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci |
    +-------+------------------------------------------------------------------------------------------------------------------------------------+

By casting to CHAR in SQL, on the python side we always have the str
that we expect here, because this is used as path attribute of a SQL
brain.

This is similar to 18deb716 (ProcessingNodeTestCase: also setRequest
in processing_node, 2023-05-02), as said in that commit message,
timerserver loop calls setRequest, but before entering the loop, we
wait for the portal to be created with:

 447   │           try:
 448   │             self.portal = self.app[self.app.test_portal_name]
 449   │           except (AttributeError, KeyError):
 450   │             continue

While accessing like this, this will load classes and initialize dynamic
modules, on python2 this was OK, but on python3 this was raising an
error because `getRequest` returned None:

  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 449, in load_module
    return self.__load_module(fullname)
  File "./parts/erp5/product/ERP5Type/dynamic/component_package.py", line 416, in __load_module
    erp5.component.ref_manager.add_module(module)
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 86, in add_module
    self.add_request(get_request())
  File "./parts/erp5/product/ERP5Type/dynamic/dynamic_module.py", line 64, in add_request
    self.setdefault(last_sync, (WeakSet(), set()))[0].add(request_obj)
  File "./lib/python3.9/_weakrefset.py", line 89, in add
    self.data.add(ref(item, self._remove))
TypeError: cannot create weak reference to 'NoneType' object

On python2, this was actually raising as well, but this error is hidden
by a `hasattr`, because on python2 `hasattr` ignores all exceptions and
on python3 it only ignores only `AttributeError`.

  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 61, in createAccessorHolder
    self.applyOnAccessorHolder(accessor_holder, expression_context, portal)
  File "./parts/erp5/product/ERP5Type/Core/PropertySheet.py", line 175, in applyOnAccessorHolder
    for property_definition in self.contentValues():
  File "./parts/erp5/product/ERP5Type/Core/Folder.py", line 1570, in contentValues
    portal_type_id_list = self._getTypesTool().listContentTypes()
  File "./parts/erp5/product/ERP5Type/Tool/TypesTool.py", line 173, in listContentTypes
    provider_value = _getOb(provider, None)
  File "./eggs/Zope-4.8.7-py2.7.egg/OFS/ObjectManager.py", line 323, in _getOb
    if id[:1] != '_' and hasattr(aq_base(self), id):
  File "./parts/erp5/product/ERP5Type/dynamic/lazy_class.py", line 120, in __getattribute__
    self.__class__.loadClass()

And simplify just a bit the code

FutureWarning: This search incorrectly ignores the root element, and will be
fixed in a future version.  If you rely on the current behaviour, change it to
'.//role'.

py3: TestTradeModelLineMixin inherited from UserDict() to store values on the class direcly (!1751).

This does not work with py3:
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 941, in main
      result = runUnitTestList(test_list=args,
    File "parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 703, in runUnitTestList
      result = TestRunner(verbosity=verbosity).run(suite)
    [...]
    File "parts/python3/lib/python3.9/unittest/runner.py", line 184, in run
      test(result)
    [...]
    File "parts/python3/lib/python3.9/unittest/suite.py", line 84, in __call__
      return self.run(*args, **kwds)
    File "parts/python3/lib/python3.9/unittest/suite.py", line 111, in run
      if _isnotsuite(test):
    File "parts/python3/lib/python3.9/unittest/suite.py", line 369, in _isnotsuite
      iter(test)
    File "parts/python3/lib/python3.9/collections/__init__.py", line 1067, in __iter__
      return iter(self.data)
  AttributeError: 'TestComplexTradeModelLineUseCaseSale' object has no attribute 'data'

Use a dedicated dict() to store these values as there was no strong reasons to
inherit from UserDict() here and this makes test implementation easier too...

group_id is used as key of OOBtree and as documented, it's not possible to mix
keys that can not be compared, so we can not have a mix of string and bytes, for
consistency with other BTrees, such as the ones used for OFS.

group_id is also used in a SQL column which is BINARY, this is problematic on
py3 because the selected values will be returned as bytes, but we expect str
here. Because we don't want to run a data migration, we adjust the select
methods to convert to str while selecting.

Since years there was a warning that id_group must be a string, now we make it a
bit stricter, we also enforce that the id_group is valid UTF-8.

A few more tests and assertions were also added.

Since Zope commit 020685087 (`Allow ZPublisher to handle a query string together
with a request body (#1124)`, 2023-05-15) Zope tries to process all XML HTTP
requests as XML-RPC and we need to tell that these SOAP requests are not XML-RPC.

py2/py3: Base64 encode inventory cache, as Shared.DC.ZRDB.DA.SQL tries to decode bytes to str (!1751).

Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

Followup of 94739085.

key()-based sorting is now used instead, available since Python 2.4.

Jupyter integration is more or less abandoned. Just do the minimum to make tests pass.

Also, remove `if True/1/False/0` statement raising pylint warnings.
Co-authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>

* Zope API changes:
  + publish():
    - stdin is now BytesIO rather than StringIO.
    - Returned value of a script is passed to str() in python2, not in python3 anymore.
  + HTTPResponse `body` property is now bytes().
  + OFS.Image.File file parameter is bytes().
  + zope.interface implements() is now @implementer decorator.
  + Python standard logging module recommended instead of zLOG.
* Python3 API changes:
  + builtin reduce() was removed.
  + urlnorm is now available.
  + Use BytesIO rather than StringIO to follow py3 API.
  + hmac.new() requires digestmod argument from Python 3.8.
  + Use six.moves library to handle moved objects from py2 to py3.
  + `modernize -f xrange_six` then slightly adjusted manually to just use range
     where it does not make a significant difference (for example in test).
  + base64.b64encode() now expects bytes().
  + UserDict() interface changed:
    - New parameter in update() and pop().
    - `failobj` setdefault parameter renamed to `default`.
  + ensure_list() on dict.{values,items}() and list(dict) for dict.keys() when
    we really need a list and not an iterable (Python3).
  + Make dict iteration works on both version of Python.
    - Use six.iter{items,values,keys}().
    - has_key() has been removed.
    - Make sure that dict.{items,values,keys}() returns a real list when
    modified (ensure_list()).
  + Comparisons between int and NoneType raises TypeError.
    + BTrees key must be str() not int() (_getOb()).
  + No more unbound methods in python3 so use six.get_unbound_function().
  + Exceptions:
    - No longer behave as sequences. Using `args` attribute instead.
    - When an exception has been assigned using `as target`, it is cleared at
      the end of the except clause.
  + file: py2 was returning `str` upon reading, now it returns text strings.
    Also, opening mode is text strings by default.
  + Data strings are bytes().
    - Replace str() by bytes().
  + iterators no longer have next() method, instead there is next() builtin.
  + New ConnectionError exception so rename existing one to not clash.
  + Integer division is now with //.
  + __nonzero__ is now __bool__.
  + apply() does not exist anymore.
  + Deprecated threading.Thread isAlive() has been removed.
  + im_func replaced by __func__.
  + Use six.with_metaclass() to define metaclass in a cross-compatible way with py2 and py3.
  + Only test method can be marked as expectedFailure(), not assert statement anymore.
  + os.path.walk() removed.
  + HTMLParser never fails: no strict mode nor HTMLParseError anymore (Python #15114).
  + Unpickler.find_global() is now Unpickler.find_class().
Co-Authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>
Co-Authored-by: Arnaud Fontaine <arnaud.fontaine@nexedi.com>
Co-Authored-by: Carlos Ramos Carreño <carlos.ramos@nexedi.com>
Co-Authored-by: Emmeline Vouriot <emmeline.vouriot@nexedi.com>

Some conversion (TextDocument) depend on the content type, so it's necessary to
first update the content type before trying to convert to base format.

Transform: Show better traceback on debugging failure of transform module import (nexedi/erp5!1751).

zope5: Since 5.8.1 missing Content-Type HTTP header is interpreted as application/x-www-form-urlencoded (nexedi/erp5!1751).

Zope.git:
  commit 5b324f6c461f5ea1cc069739b6c32a1a5ff59df9
  Date:   Thu Jan 19 07:15:18 2023 +0100
    replace `cgi.FieldStorage` by `multipart` (#1094)
    * interpret a missing `CONTENT_TYPE` as `application/x-www-form-urlencoded`

With cgi.FieldStorage, it was interpretated as text/plain so Content-Type has to
be provided now (even for WebDAV despite RFC 4918 stating that it *SHOULD* be
provided).

Some Unit Tests did not provide such header at all but this was wrong (such as
erp5_stripe:testStripe where the real request has application/json as
Content-Type and not text/plain).

See merge request nexedi/erp5!1988

to test the translation of the editor's "Maximize" button