In this specific case, the number of grouped messages is bound,
and actually not greater than the number of grouped messages
being processed for normal indexing.

The goal is to make better use of the ZODB Storage cache. It is common to do
processing on a data set in several sequential transactions: in such case, by
continuing execution of these messages on the same node, data is loaded from
ZODB only once. Without this, and if there are many other messages to process,
processing always continue on a random node, causing much more load from ZODB.

To prevent nodes from having too much work to do, or too little compared to
other nodes, this new parameter is only a hint for CMFActivity. It remains
possible for a node to execute a message that was intended for another node.

Before this commit, a processing node selects the first message(s) according to
the following ordering:

  priority, date

and now:

  priority, node_preference, date

where node_preference is:

  -1 -> same node
   0 -> no preferred node
   1 -> another node

The implementation is tricky for 2 reasons:
- MariaDB can't order this way in a single simple query, so we have 1
  subquery for each case, potentially getting 3 times the wanted maximum of
  messages, then order/filter on the resulting union.
- MariaDB also can't filter efficiently messages for other nodes, so the 3rd
  subquery returns messages for any node, potentially duplicating results from
  the first 2 subqueries. This works because they'll be ordered last.
  Unfortunately, this requires extra indices.

In any case, message reservation must be very efficient, or MariaDB deadlocks
quickly happen, and locking an activity table during reservation reduces
parallelism too much.

In addition to better cache efficiency, this new feature can be used as a
workaround for a bug affecting serialiation_tag, causing IntegrityError when
reindexing many new objects. If you have 2 recursive reindexations for both a
document and one of its lines, and if you have so many messages than grouping
is split between these 2 messages, then you end up with 2 nodes indexing the
same line in parallel: for some tables, the pattern DELETE+INSERT conflicts
since InnoDB does not take any lock when deleting a non-existent row.

If you have many activities creating such documents, you can combine with
grouping and appropriate priority to make sure that such pair of messages won't
be executed on different nodes, except maybe at the end (when there's no
document to create anymore; then activity reexecution may be enough).
For example:

  from Products.CMFActivity.ActivityTool import getCurrentNode
  portal.setPlacelessDefaultReindexParameters(
    activate_kw={'node': 'same', 'priority': priority},
    group_id=getCurrentNode())

where `priority` is the same as the activity containing the above code, which
can also use grouping without increasing the probability of IntegrityError.

This fixes the issue that a transaction with many big messages failed to
commit. By dynamically find the maximum allowed size of a query, it also
speeds up insertion by minimizing the number of queries.

This moves the remaining DTML queries to Python, dropping the 'activity' skin.

Dealing with conflicts of uids is easier if the inserted uids are consecutive:
now, only 1 random value is generated, as base uid. This also preserves the
order of insertion, which is wanted for performance reasons:
- No more random write in the primary index.
- When modifying several lines of several documents, 1 document being processed
  at a time, we'd like that any grouped activity (usually indexation) follows
  the same order, so that a processing node prefer many lines from a few
  documents instead of mixing lines from too many documents at the same time.
  This is usually better for caches.

Average age of activities is dropped because it would become
too complicated to implement and it's useless information.

The original goal was to improve performance by removing the
`processing_node_processing` index and the queries that modified
these 2 useless columns.

The root call to getExecutableMessageList (i.e. the one from distribute)
is fast enough and won't hold old revisions of the database for too long.
It is also completely read-only so it won't lock anything.

This caused useless communication with the server.

As shown in the following example, on a big catalog table,
MariaDB is able to use several indices at the same time
('...' are obfuscated unique values):

  > analyze select SQL_NO_CACHE uid, relative_url from catalog where reference='...' OR relative_url='...';
  +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
  | id   | select_type | table   | type        | possible_keys          | key                    | key_len | ref  | rows | r_rows | filtered | r_filtered | Extra                                            |
  +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
  |    1 | SIMPLE      | catalog | index_merge | Reference,relative_url | Reference,relative_url | 768,767 | NULL |    2 |   2.00 |   100.00 |     100.00 | Using union(Reference,relative_url); Using where |
  +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
  1 row in set (0.00 sec)

So mixing different dependency types with OR should be fine
(no need to split into more subqueries and join with UNION).

This fixes:

  Traceback (innermost last):
    Module Products.CMFActivity.ActivityTool, line 1373, in invokeGroup
      traverse(method_id)(expanded_object_list)
    Module Products.ERP5Catalog.CatalogTool, line 946, in catalogObjectList
      super(CatalogTool, self).catalogObjectList(tmp_object_list, **m.kw)
    Module Products.ZSQLCatalog.ZSQLCatalog, line 813, in catalogObjectList
      **kw
  TypeError: catalogObjectList() got an unexpected keyword argument 'group_id'

/reviewed-on nexedi/erp5!819

* Set MAKEFLAGS with -l to limit number of jobs by load and number of
processes
* set NPY_NUM_BUILD_JOBS & BUNDLE_JOBS to `node_quantity` instance
parameter

/reviewed-on !809

Originally, uids somehow sorted messages by date of insertion, in particular
for those that were created within the same second. But since random uids,
such messages became validated or processed in random order.

Note however that by default, messages created in the same transaction all have
exactly the same date, so commit a42da4de
("CMFActivity: Do not use offset for scanning messages to validate.")
forces us to keep the ordering on uids (in addition to priority/date).

Existing instances will upgrade automatically, using the already existing code
to upgrade tables in a generic way. You should see the following logs:

    INFO CMFActivity 'message_queue' table upgraded
    ALTER TABLE message_queue
      MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
      MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing
    INFO CMFActivity 'message_job' table upgraded
    ALTER TABLE message_job
      MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
      MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing
    INFO CMFActivity 'message' table upgraded
    ALTER TABLE message
      MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
      MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing


/reviewed-on nexedi/erp5!820

so that options are kept

Same behaviour than xhtml style

Next step is to activate more contextual listbox actions.

Move listbox select button to the right, to simplify cancellation.

WebSite redirection must be tested against different virtual host monster configuration
TODO: No idea how to fetch the ipv6 in live test

https://lab.nexedi.com/nexedi/slapos/tree/master/software/proftpd/test
was just named "test", we want the test result to be named "proftpd"
instead.

/reviewed-on nexedi/erp5!815

See nexedi/erp5@04939506

Since recent rework of isSubtreeIndexable, direct indexation of a trash bin
is fixed, but recursive indexation recursion condition broke: it stop just
above the trash bin instead of stopping just below it.

Revert "mark file uploading tests as expected failure." as these Functional Tests are not supposed to fail.

This reverts commit ade16831.

enablePrivilege, and thus UniversalFileRead used to upload files, was disabled
in Firefox 17. Since SlapOS Firefox has been upgraded, "The operation is insecure"
error is raised when uploading files. However, this doesn't mean that these tests
should be expected to fail (and if they are, they should be removed instead).

This workflow involved an Assignee who can open, close, re-open and an
Assignor who can close definitively.
This is usually configured so that accountants are Assignee and CFO is
Assignor.

We realized that re-opening a Period that was previously closed is
something we don't want the accountants to do without CFO's approval.

To support this configuration, we only allow Assignor to re-open.

Now Assignee can open and close temporarily and Assignor can re-open and
close definitively.

/reviewed-on nexedi/erp5!813

This is really a hack, which may be integrated into renderJS directly

Reduce potential Zelenium test timeout by waiting for iframe to be loaded one by one

This check got lost when the first isSubtreeIndexable call was done on
parent document and not on self.

This partially reverts:

commit 76e3c115
Author: Vincent Pelletier <vincent@nexedi.com>
Date:   Mon Dec 10 16:40:48 2018 +0900

    Base: Fix isAncestryIndexable implementation.

as it accidentally carried over a totally unrelated (and unfinished)
change.