See merge request nexedi/erp5!1671

a modified version from $366
that supports patterns like
DateTime(2001, 02, 03) => DateTime(2001, 2, 3)
and not DateTime(2001, 0o2, 0o3)

Javascript does not support big integers, so the backend have to be
careful not to return too large numbers in the json. Nowadays there
is bigint, but not equivalent for decimal numbers, so the backend
tries to detect if numbers are too big and returns them as string
in that case - which is better than returning a different number
because it displays correctly in the result table. The pivot table
will still wrong, but at least the table of results is OK.

use sax.xml API which gives us the first parser available, instead of
using a specific parser that happens to come from pyxml

serialize as json instead of using xml.marshal.generic which was
provided by PyXML

See merge request nexedi/erp5!1668

See merge request nexedi/erp5!1665

This allows custom movement generators to set ad-hoc properties on
generated movements depending on properties of the corresponding
input movement.

There was already _getUpdatePropertyDict which is similar, but was
sometimes not enough because it can only be use to set extra
properties on all movements.

As a result, when we needed a rule with custom properties on
generated movements, we resorted to overriding getGeneratedMovementList
and duplicating the full logic of the method. Existing cases have
been updated to use the new _updateGeneratedMovementList and in the
case of InventoryAssetPriceAccountingRuleMovementGenerator we no
longer need to duplicate the logic.

Also remove the comment suggesting overriding getGeneratedMovementList
in subclasses, now several "extension" methods exists, so it should
not be needed to override getGeneratedMovementList, every needs
should be coverred by defining _updateGeneratedMovementList or
_getUpdatePropertyDict

This patch does not seem necessary

This SELECT option was added when the UPDATE query was split into a
SELECT + UPDATE query pair, in order to try to avoid deadlocks in mariadb.
  commit 7daaf0a5
  Author: Vincent Pelletier <vincent@nexedi.com>
  Date:   Fri Jun 7 16:53:11 2013 +0200

      Work around poor UPDATE use of index.
That was not completely successful, and another attempt was since made,
which further ipmproved the situation:
  commit e21aa47f601a98f767e2d3e8490b20b7f409933c
  Author: Vincent Pelletier <vincent@nexedi.com>
  Date:   Fri Sep 17 11:34:46 2021 +0900

      CMFActivity.Activity.SQLBase: Reduce the number of deadlocks

This change made the "FOR UPDATE" option obsolete: there should be
virtually no parallelism between the SELECT and UPDATE queries.
By which I mean only 3rd-party accesses (like an admin on a mysql console)
would bypass the SQL lock, causing index-update-level locking issues.

Checking now, I see that removing "FOR UPDATE" on the "node>0" query makes
that query (with LIMIT 30) run about 5 times faster:
With FOR UPDATE:    0.509s
Without FOR UPDATE: 0.090s
This difference is enough to cause significant activity starvation on a
63-processing-nodes cluster where such activities run in about 1s, just
because of the amount of contention on the SQL lock, itself because of the
duration of every execution of this query.

So remove this now-redundant option.

It is expected that this will improve activity processing performance in
general, beyond just the starvation issue which led to this discovery.

Draft: The first state when we create a new Sale Opportunity.

Open: All information of the Sale Opportunity is entered such as prospect's contact email, questions, etc

Suspended: We replied to prospect, waiting for their return.

Closed: The Sale opportunity is either 1) accepted by prospect, turn into a Sale Order; or 2) Clearly rejected by prospect.

Expired: No return/ news from prospect for some time.

Deleted: in case added by mistake, so user can delete it

See merge request !1497

Reference must always be after the web section path.

This will also forbid using document relative url directly (web_page_module/123).

See merge request nexedi/erp5!1666

For example, Web Section can contain a Script (Python) to set Access
Rule, but the parent section's objectValues(portal_type='Web Section')
should not include such object.

When there are many simultaneously-pending activities attached to any
processing node family, the
  node>=0
subquery becomes dominant (taking hundreds of time longer than the other
subqueries). As a consequence, this starves processing nodes of activities
and increases the CPU needs of the mariadb process hosting the activity
tables.
So, move this subquery out of the regular codepath, and only run it if no
other subquery found any activity:
- there is no activity preferentially targeting the current node
- there is no activity bound to any of the current node's families
- there is no activity without any node preference at all
Also, simplify the content of that subquery: the effective priority can
only be 3 * priority + 1 when this query is run, and node=0 rows can be
excluded (they should not exist in the current database view).
Also, factorise the logic producing "node=processing_node" and
"node IN node_set" subqueries, for simplicity. In turn, this makes all
family-dependent subqueries use a simple equality test, ensuring a stable
query plan independently from the number of families the current node is
member of.
Also, use "UNION ALL" always, as now:
- all subqueries have stritly distinct result sets
- as per mariadb documentation, "UNION [DISTINCT] applies to all UNIONs
  on the left", so the original comment about where ALL is used was
  incorrect in assuming it was improving the effective query performance
Also, line-split SQL queries as visible in the python source to be more
readable, without effect on the produced SQL.
Also, line-split a few non-trivial python expression to make their
internal structure immediately apparent.

Another effect of this change this change is to reduce activity theft
(activities to be preferentially executed by one node being executed by
another), potentially improving object cache hit-rate and hence decreasing
I/O pressure on the ZODB.

this way they will be expanded in simulation

this will have a side effect that Trade Model Path will also consider
destination carrier, like it does with source_carrier.

Not having destination carrier seems not intentional, during the
early days of "new simulation", source_carrier was added but not
destination_carrier, this was commit d487196a (A first idea of how
BusinessPath will look at the end, 2010-05-22)

Instead of hardcoding categories group in TradeModelPath, introduce
two new groups (as planned in the XXX comments) to list source
and destination arrow base categories.

See merge request !1655

With a fix for the URLs query string being escaped twice.

Also fixes icons not cached in sercice worker for OfficeJS.


See merge request nexedi/erp5!1661

this change was made with:

    find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/\\?t=445cf24ebd//g'

This timestamp break officejs offline capabilities.

This reaplies 40f6c8fe ([erp5_xhtml_style] ckeditor: drop hardcoded
timestamp parameter in URL, 2017-10-16) on the updated CKEditor

This was done with:

    find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/"M6K9"/""/g'

M6K9 was the timestamp for this build, that we can see at the top of
ckeditor.js

done with an external method to upload the content of zip file in
skin folder.

    import zipfile
    from io import BytesIO

    def uploadZip(self):
      dest = self.getPortalObject().portal_skins.custom
      with zipfile.ZipFile(
          '/srv/slapgrid/slappart3/srv/project/erp5/ckeditor_4.19.1_4b98d281bc34.zip'
      ) as f:
        for m in f.namelist():
          current_dest = dest
          print(m)
          for part in m.split('/')[:-1]:
            if part not in current_dest.objectIds():
              current_dest.manage_addProduct['OFSP'].manage_addFolder(id=part)
            current_dest = current_dest[part]
          fname = m.split('/')[-1]
          if fname == '_translationstatus.txt':
            fname = 'translationstatus.txt'
          current_dest.manage_addProduct['OFSP'].manage_addFile(
            id=fname, file=BytesIO(f.read(m)))
      return repr(dest)

Using make_query makes sure that the query parameters are properly
encoded and also change selection_index to become an int, because
it keeps the type of parameters.

As a consequence, we had to adjust a few place in the code where
selection_index was tested for truthiness: because "0" as a string is
true, but 0 as an int is not. For that, we changed to test the presence
of selection_name instead of testing selection_index, as they are always
used together.

This fixes a problem that & in URL was encoded twice for listbox
anchor links ( bug_module/1137 )

02011d8e (immediateReindexObject: use super user to reindex script,
2015-12-14) did not apply for inventory, because they were overloading
immediateReindexObject.
Introduce a new level, _immediateReindexObject that will hold the actual
reindexing logic.

Previously this method was using PortalContent.reindexObject which was
monkey patched, to make things less complex and more future proof, move
the monkey patch to a method on base class.

This also drops alternateReindexObject on BalanceTransaction, because it
is already defined in Inventory

This is supposed to fix "Should not load state for ${oid of a skin} when
the connection is closed" sometimes happening with live tests (especially
when the test self.publish and the developer access the site while the
test is suspended on a debugger breakpoint).

The object accessed after the connection is closed was a skin (python
script, sometimes page template or form) that was cached in SKINDATA.
The mechanism to prune entries from the cache uses REQUEST.hold API
which expects that REQUEST.close is called on request, but because
requests were not closed at the end of the request, it happened that
the cache was reused from another connection.

This change to close the requests, like ERP5TypeTestCase is doing in
tearDown (the actual close is done by Testing.ZopeTestCase.connections).

By closing requests at the end of tests, we also have to change so that
at the beginning of the test we initialize the request, by using the same
setSite and setupCurrentSkin that are done in ERP5TypeTestCase.

SKINDATA is an implementation detail that should not be accessed from
this level.

In SkinTemplateItem.install, self._objects contains entries for skin
folders and for all skins. objectValues method calls was called for skin
folders (as expected) and also for all skins, which acquire objectValues
from skin folder and do the work again.

This simplifies this by only running this for skin folders.

"default" URL is the URL when listbox column does not use URL column
and when the brains do not have a getListItemUrl method.

Move the computation in a lazyMethod, so that it is computed only once
per line instead of once per cell.

Also remove a try/except, I don't think this code is supposed to get an
AttributeError