Notably showned as a companion of the editor gadget

WebSection.getDocumentValueList() returns absolutely all the documents (as catalog brains) reachable by the web section. Then, calling .getUid() on them load all the objects in memory. The number of documents is of the order of hundreds, even on a small ERP5 (web pages, web scripts, etc.).

Limit the documents to retrieve in WebSection_getLatestDiscussionPostList by filtering on the portal_type, and do not load them needlessly in memory.

  This aims to add compatibility with BTreeFolder2 API, even it is not
  required. Since some checkConsistency may call self._cleanup() regardless
  expecting that the folder is a [H]BTreeFolder2 always.

  This was detected when a post upgrade constrant was included to
  portal_categories

The methods used in indexing did not make a difference between the case
where the price is None (ie. price is not set) or where the price is
set to 0 - in both cases this was saved as NULL in stock.total_price
column. This is incorrect, we need to keep the distinction between
these two cases also for inventory calculation.
We had some places where we select IFNULL(stock.total_price, 0) to work
around this, we don't plan to change the existing ones for now, but
while discussing on !1974
we concluded that a newly idenfified case of a problem consequence of
these NULL should be handled by fixing the indexation. To benefit from
the fix, impacted instances will have to reindex documents present in
the stock table with stock.total_price is null.

__import__ `fromlist` argument was wrong. It was working anyway with Python2 but
not anymore with Python3, raising a `ModuleNotFoundError` exception. According
to Python `__import__(name, globals, locals, fromlist)` documentation:

  When the `name` variable is of the form `package.module`, normally, the
  top-level package (the `name` up till the first dot) is returned, *not* the
  module named by `name`. However, when a non-empty `fromlist` argument is
  given, the module named by `name` is returned.

Thus, the following patterns were wrong:
  * __import__(MODULE_NAME, globals(), locals(), MODULE_NAME)
    => Iterate through each character of MODULE_NAME as fromlist is expected to
       be a list/tuple.
  * __import__(MODULE_NAME, globals(), locals(), [MODULE_NAME])
    => This works but actually tries to import MODULE_NAME object from
       MODULE_NAME module (no error if it cannot).

The goal of such __import__ calls were for __import__ to return the right-end
module instead of the top-level package. In such case, `fromlist=['']` is the
way to go as it __import__ does not check if the object exists in the module if
it's an empty string. However, it is even better and easier to read to use
importlib.import_module() for that...

Also, add `from __future__ import absolute_import` because python2 tries both
relative and absolute import (level=-1 __import__ parameter) whereas python3
does absolute import by default (level=0).
Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

When under a pinDateTime context, d.isPast() was comparing d with the
actual current date, not the fake date from the pinned context.

Also, make the methods class methods, so that they can be used as
external methods in Zelenium tests.

 - Products.PythonScripts now has a __loader__
 - do not use < > as filename for component on py3, this makes linecache
 work out of the box. On py2 I think it was causing errors trying to
 actually open the file, but this does not seem needed on py3 and
 simplifies everything

new test with ERP5-style workflow to cover python3 migration

with ROUND_DOWN, we have a different behaviour between py2/py3, that caused failures in
erp5_simplified_invoicing:testTradeModelLine.

Here is what happened in _round(1.9999999999999998) in
bt5/erp5_simulation/DocumentTemplateItem/portal_components/document.erp5.FloatEquivalenceTester.py

* py2
decimal.Decimal(str(1.9999999999999998)).quantize(decimal.Decimal('0.000001'), 'ROUND_DOWN')
=> Decimal('2.000000')
(because str(1.9999999999999998) is '2.0')

* py3
decimal.Decimal(str(1.9999999999999998)).quantize(decimal.Decimal('0.000001'), 'ROUND_DOWN')
=> Decimal('1.999999')
(because str(1.9999999999999998) is '1.9999999999999998')

But ROUND_DOWN result of 1.9999999999999998 with 0.000001 precision should be 1.999999 thus py2 behaviour is wrong.

Run zope with ERP5_COMPONENT_OVERRIDE_PATH set to a : separated list of
path containing components files to load components from files instead
of portal_components.

This can be used to repair zope after a bad edit of component.

for pylint 3 to understand erp5 dynamic modules

We want these columns to be "binary" so that mariadbd compares them
without any collation, but we don't want to use binary type, because on
python 3 the brain attributes are bytes (user ids are strings everywhere
else).

This also normalize the same columns present in worklist cache table.

I don't know a PYTHONHASHSEED to see the problem on python2, but on
python 3 erp5_payroll_l10n_fr:testSimplifiedPayslipReport fails all the
time with this.

On python3 the percentages are sometimes printed as 6.8500000000000005%

cookie crumbler uses username:password base64 encoded and this relies
on extracting the username from the cookie to set the session, but
other authentication methods do not embed the username, but still share
the same scripts to set the authentication cookie (so that we have one
central point setting cookie with all the necessary attributes for
security).
The problem is that it's incorrect to try to decode an username with
cookies from other authentication methods, simply because they do not
contain the username. On python2 this was not causing visible error
because everything is str, but the decoding error on python3 revealed
this.

in Python 2, _aq_dynamic() returns None without try..except but it raises ValueError in Python 3.

(python 2)
> /SR/parts/erp5/product/ZSQLCatalog/Extensions/zsqlbrain.py(31)_aq_dynamic()
 31  ->   def _aq_dynamic(self, name):
 32         """Acquire an attribute from a real object.
 33         """
 34         if name.startswith('__') :
 35           return None
 36         return getattr(self.getObject(), name, None)
((Pdb)) getattr(self.getObject(), name, None)
*** ValueError: Unable to getObject from ZSQLBrain if ZSQL Method does not retrieve the `path` column from catalog table.
((Pdb)) r
--Return--
> /SR/parts/erp5/product/ZSQLCatalog/Extensions/zsqlbrain.py(36)_aq_dynamic()->None # <-- !!!
Co-authored-by: Jérome Perrin <jerome@nexedi.com>

- HTMLParseError no longer exist, on python3 parse_declaration throws
AttributeError

py2:
  https://github.com/python/cpython/blob/2.7/Lib/markupbase.py#L135-L140
  https://github.com/python/cpython/blob/2.7/Lib/HTMLParser.py#L124
py3:
  https://github.com/python/cpython/blob/3.12/Lib/_markupbase.py#L130-L134

- scrubHTML must pass `html` as unicode on python2 and str on python3,
adjust the check to cover both py2 / py3

On python2, attributes of new style classes (classes, not instances)
were not properly validated. On python3, attributes of classes were not.

In current versions, sorted is allowed in RestrictedPython.

Also, our implementation of sorted was not correct with generators,
it consumed the generator by iterating on it a first time to check the
values.

otherwise we will have an Exception in Python 3 while importing a Business Template,
if a new portal type and its documents exist in the same Business Template.

while importing a content file, like *.js...

(BusinessTemplate.py)
  try:
    setattr(obj, property_name, data)
  except BrokenModified:
    obj.__Broken_state__[property_name] = data # <-- !!!
    obj._p_changed = 1

obj.__Broken_state__ access here also raises BrokenModified.

while importing an XML file...

(ZODB/broken.py)
  def __new__(class_, *args):
    result = object.__new__(class_) # <-- !!!
    result.__dict__['__Broken_newargs__'] = args
    return result

we get an exception 'TypeError: object.__new__(Portal Type) is not safe, use Base.__new__()'
Co-authored-by: Kazuhiko SHIOZAKI <kazuhiko@nexedi.com>

  This prevents be confuse while editing the Connector.

  This allow user know what is he configuring or accessing

One problem with hexdigest is that it's longer and the column was
created as binary(16). We don't have a mechanism to run migrations on
this table, so we use UNHEX in SQL to have the equivalent as digest()
from hexdigest()
Co-authored-by: Jérome Perrin <jerome@nexedi.com>