See merge request nexedi/erp5!1553

Some combinations of periodicity, for example repeat every first week
of the year and every month February are impossible (because the first
week of the year is always in January) and such configurations caused
infinite loops or probably overflow if we wait long enough.

The algorithm being to try the next day until all constraints are met,
it is not guaranteed to terminate.

To make sure the algorithm terminate, we rely on the fact that calendars
repeat after some time, so if after a few years we did not find a
matching combination, we can stop retrying.

according to https://en.wikipedia.org/wiki/Determination_of_the_day_of_the_week

> Each leap year repeats once every 28 years, and every common year
> repeats once every 6 years and twice every 11 years.

so trying for 28 years should be enough to see all combinations

In ZODB 5, with commit b6ac40f1 (Uses an unwrapped transaction manager,
2018-10-14) the transaction is bound to the thread opening the
connection.
The previous pattern of opening transaction in the main thread and
passing the already-open connection to the working thread caused the
working thread connection to be managed by the main thread connection
and in ZODB 5 cause the test to block.

Fix this by passing a connection factory method and opening
connection in working thread.
Also simplify closing of connection by using a closing context manager.

With upcoming ZODB 5, oids (used as persistent references in pickles)
are no longer str as it use to be with ZODB 4, but instances of
zodbpickle.binary, which with zodbpickle 1 are a subclass of str on
python2.

OrderedPickler was a subclass of pickle.Pickler, the pickler from standard
library, but this pickler was not able to use a str subclass for persistent
references, when pickles are loaded with noload method, persistent_load
is called with `None` instead of the actual string subclass instance.
This was problematic in the XMLExportImport handling of business templates,
because ZODB.serialize.referencesf was unable to find persistent references.
The error was:

    ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 664, in referencesf
        assert isinstance(reference, list)
    AssertionError

because the reference was None.

zodbpickle 2 changed to make zodbpickle.binary implemented in C, which
was failing earlier, because pickle.Pickle can not pickle these objects,
failing in an error like this:

    lib/python2.7/copy_reg.py", line 70, in _reduce_ex
        raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle binary objects

This change also simplify our own implementation, by dropping jython
support and calling save_dict on the super class instead of copying the
implementation.

Further references:

- minimal script to reproduce the issues:

```python
from __future__ import print_function
import io
import pickle

import zodbpickle
import zodbpickle.pickle
import zodbpickle.fastpickle

class ExternalObject(object):
  def __init__(self, oid):
    self.oid = oid

def persistent_id(obj):
  if isinstance(obj, ExternalObject):
    return obj.oid

def persistent_load(persid):
  print('persistent_load called with persid', repr(persid))

o = ExternalObject(oid=zodbpickle.binary("binary persid"))

for pickler_class in pickle.Pickler, zodbpickle.pickle.Pickler:

  f = io.BytesIO()
  p = pickler_class(f, 1)
  p.persistent_id = persistent_id
  p.dump(o)

  print('dump with pickler %s:\n  %r' % (pickler_class, f.getvalue()))

  # ZODB uses this unpickler
  up = zodbpickle.fastpickle.Unpickler(io.BytesIO(f.getvalue()))
  up.persistent_load = persistent_load
  up.noload()
```

```console
$ python2 repro.py # with zodbpickle 1
dump with pickler pickle.Pickler:
  'ccopy_reg\n_reconstructor\nq\x00(czodbpickle\nbinary\nq\x01c__builtin__\nstr\nq\x02U\rbinary persidq\x03tq\x04Rq\x05Q.'
persistent_load called with persid None
dump with pickler zodbpickle.pickle_2.Pickler:
  'U\rbinary persidq\x00Q.'
persistent_load called with persid 'binary persid'
```

```console
$ python2 repro.py # with zodbpickle 2
Traceback (most recent call last):
  File "repro.py", line 45, in <module>
    p.dump(o)
  File ".../lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File ".../lib/python2.7/pickle.py", line 273, in save
    self.save_pers(pid)
  File ".../lib/python2.7/pickle.py", line 340, in save_pers
    self.save(pid)
  File ".../lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File ".../lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle binary objects
```

* ZODB change starting to use zodbpickle.binary instead of str:
12ee41c4 (-ZODB now uses pickle protocol 3 for both Python 2 and Python 3., 2018-03-26)
Since of 5.4.0 release

* zodbpickle change starting to use C objects for zodbpickle.binary:
bbef98c (Implement zodbpickle.binary in C for Py27., 2019-11-12)
Since of 2.0.0 release

Now that we fail immediately in case of failure, the deadline can be
safely increased, because it only protects against kind of infinite loops.

Increasing the delay should fix RuntimeError: tic is looping forever
errors with only messages in status -1, that we sometimes saw on testnodes.

Now that tic retries until the deadline is reached or all messages has
failed, it can lead to situations where developer have to wait until the
deadline, when a message failed but other messages (typically scheduled
to run after the failed message) were still running.

By stopping as soon as one message is failed, in this scenario the
developer does not need to wait until the deadline.

1. don't manipulate data base to avoid resend to web service in case of data base conflict
2. use existing property
3. change alarm frequency

In our use, waitress is embedded in Zope and not as a proxy, so it has no
legitimacy emitting Via headers.
While this process often is the origin server, Server response header is
optional and should be left to the lower layers to control, and not
auto-generated by waitress.

See merge request !1500

1. display employee number
2. add preview
3. add parameter to send to maileva

maileva

See merge request !1544

erp5_api_style: Use default module for object creation

erp5_api_style: jIOStyle fix error handling for post

erp5_api_style: Start improving error feedback

erp5_api_style: Add error record module and portal type

erp5_api_style: Fix error output as jIO Style

erp5_api_style: Introduce jIO Web Section

erp5_api_style: catch value Error when error text is not a json

erp5_api_style: Add generic way to handle errors

erp5_api_style: Fix Web Section Portal Type for api style

erp5_api_style: Fix jIO Api Style

* use self.REQUEST instead of parameter in jIO Web Section
* Use List error in allDocs

erp5_api_style: API error return http error code 400

erp5_api_style: Display actions title in jio web section configuration listbox

erp5_api_style: allDocs return indented result

erp5_api_style: Add response schema to jIO Style. Object creation return 201 http code

erp5_api_style: Post set status to 201 only if status is not yet defined

erp5_api_style: action list is now sorted by float index

erp5_api_style: all docs return error dict like post

erp5_api_style: error can return error name and error link

erp5_api_style: improve error handling in jIO Style

* Give name to errors
* Error Name is stored in error record title

erp5_api_style: Add views for error records

erp5_api_style: add script to provide JSON API Hyperschema

erp5_api_style: fixup not found document return 404 error

erp5_api_style: api is in charge of JSON decoding and provide utf 8 data

erp5_json_form: Improve error feedback

erp5_json_form: Add missing property sheet JSON Form

erp5_json_form: Add response Schema to JSON Form

erp5_json_form: Add Init Script on JSON Form

* Useful to set content type to "application/json"

erp5_json_form: Cleanup JSON Form document

* Add basic property sheets
* Add init script to set Constraint and Type
* Validator can return the list of errors
* Provide field library

When EmptyCriterionValid property is true, this method is expected to
return a query which does not match any document. This only happens when
query_list is empty, but because of category membership checking,
query_list is never empty: it at least contains two ComplexQueries which
themselves may contain an empty query list, and which match all documents.
Calling getCategoryParameterDict with an empty list is dubious, but
changing its behaviour in ZSQLCatalog may affect more than just predicates,
so change the behaviour in Predicate class directly by checking whether
there is any category being matched to begin with.

Pygolang installs import hooks for pytest and ipython, to add
exception-chaining support into them(*) _iff_ (if and only if) those
modules are actually used. This works via Importing[1] which
pre-installs artificial modules into sys.modules that catch
__getattribute__ and try to import corresponding module for real on
first access. Usually everything is fine.

But with pylint/astroid, if the checker code happens to run with those
LazyModules installed, and the checked code has `import sys` somewhere,
astroid eventually delves into processing sys, then sys.modules and
wants to represent that sys.modules dict as dict of constant. Then, when
e.g. sys.modules['_pytest'] is processed, corresponding module object is
checked for .__class__, which raises ImportError if pytest is not
actually available:

    ( https://erp5js.nexedi.net/#/test_result_module/20220127-129289AE2/33 )
    ...
    File ".../eggs/astroid-1.3.8-py2.7.egg/astroid/node_classes.py", line 553, in __init__
        for k, v in items.items()]
      File ".../eggs/astroid-1.3.8-py2.7.egg/astroid/node_classes.py", line 962, in const_factory
        return CONST_CLS[value.__class__](value)
      File ".../eggs/Importing-1.10-py2.7.egg/peak/util/imports.py", line 254, in __getattribute__
        _loadModule(self)
      File ".../eggs/Importing-1.10-py2.7.egg/peak/util/imports.py", line 222, in _loadModule
        reload(module)
    ImportError: No module named _pytest

-> Fix it by detecting those lazy modules and not letting them go through
normal const_factory not to crash.

/cc @jerome, @arnau
/reviewed-by @rafael
/reviewed-on nexedi/erp5!1546 and nexedi/slapos!1117

[1] https://pypi.org/project/Importing/

(*) see:
    https://lab.nexedi.com/nexedi/pygolang/blob/pygolang-0.1-0-g7b72d41/golang/_patch/__init__.py
    https://lab.nexedi.com/nexedi/pygolang/blob/pygolang-0.1-0-g7b72d41/golang/_patch/pytest_py2.py#L48-51
    https://lab.nexedi.com/nexedi/pygolang/blob/pygolang-0.1-0-g7b72d41/golang/_patch/ipython_py2.py#L45-48

See merge request nexedi/erp5!1517