In nexedi/erp5!225 we realized google gadgets ( calendar, clock, maps and search ) no longer work since years, since they were hosted on google servers and they are longer hosted there.

Gadget framework is still working, our own gadgets (persons, worklists etc) too, in this merge request we just remove the gadgets that were hosted on google and not longer work.

I believe they were kind of example and we should not reimplement them or try to replace them.

/cc @gabriel @Tyagov @romain @kazuhiko 


For more reference https://support.google.com/websearch/answer/2664197?hl=en  
https://en.wikipedia.org/wiki/IGoogle 
 https://en.wikipedia.org/wiki/Google_Gadgets 


/reviewed-on nexedi/erp5!255

For years, erp5_calendar was keeping a duplicated copy of [`z_catalog_stock_list`](https://lab.nexedi.com/nexedi/erp5/blob/985bbc0f716397c899e6af69ef431db86923ab34/product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_catalog_stock_list.sql). In a79ca4c1, the functionality was merged in erp5_mysql_innodb and the duplicated copy was removed from erp5_calendar.

Problem is that when we update erp5_calendar, `z_catalog_stock_list` is removed.

We discussed in a79ca4c1 (comment 15577) and suggested to use *Path of objects that should be kept* like we already did in some other business templates.

I tried, but it did not work, because [`CatalogMethod.preinstall`](https://lab.nexedi.com/nexedi/erp5/blob/985bbc0f716397c899e6af69ef431db86923ab34/product/ERP5/Document/BusinessTemplate.py#L2995)  overwrites the returned value of [`ObjectTemplateItem.preinstall`](https://lab.nexedi.com/nexedi/erp5/blob/985bbc0f716397c899e6af69ef431db86923ab34/product/ERP5/Document/BusinessTemplate.py#L1117)  (which supports template_keep_path_list) with the returned value of [`BaseTemplateItem.preinstall`](https://lab.nexedi.com/nexedi/erp5/blob/985bbc0f716397c899e6af69ef431db86923ab34/product/ERP5/Document/BusinessTemplate.py#L485) which does not support it.

My understanding is that the former is necessary to install the z sql methods and later to install the configuration on the catalog
 ( see efe5294e ), but the order does not really mater, so it's safe to give priority to `ObjectTemplateItem.preinstall`.

I also add some tests for template_keep_path_list, I think it was not tested, and modified erp5_calendar as suggested.

/cc @seb @kazuhiko @aurel @tiwariayush @gabriel 

/reviewed-on !260

Payroll related document classes (AnnotationLine,  PaySheetLine, PaySheetTransactionLine, PaySheetCell) are not different from Invoice-related classes.

This work aims to remove these document classes and base the portal types on the generic invoice classes.

This work started after finding out that Pay Sheet Lines were appearing in the "Account Statement" report, on side of Pay Sheet Transaction Lines. This happened as PaySheetLines is overiding the "isAccountable" method to return True. This merge request, besides removing duplicate code, fixes the accountable property of Pay Sheet Lines, and updates the call to the inventory API to generate correct accounting/payroll reports.

/reviewed-on nexedi/erp5!223

93e30e5e introduce a  new `user` table that is listed as a *search table*, so until this table is created all catalog queries are failing, and alarm tool, which relies on catalog fail with such a traceback:

```
ERROR TimerService Process timer error
Traceback (most recent call last):
  File "parts/erp5/product/TimerService/TimerService.py", line 102, in process_timer
    DateTime(prev_tick), DateTime(next_tick))
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 175, in process_timer
    self.tic()
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 135, in tic
    for alarm in self.getAlarmList(to_active=1):
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 111, in getAlarmList
    alarm_date={'query':now,'range':'ngt'}
  File "parts/erp5/product/ERP5Catalog/CatalogTool.py", line 702, in unrestrictedSearchResults
    return ZCatalog.searchResults(self, **kw)
  File "parts/erp5/product/ZSQLCatalog/ZSQLCatalog.py", line 1091, in searchResults
    return catalog.searchResults(REQUEST, **kw)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2585, in searchResults
    **kw
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2554, in queryResults
    **kw
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2418, in buildSQLQuery
    ignore_unknown_columns=ignore_unknown_columns,
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2394, in buildEntireQuery
    query=self.buildQuery(kw, ignore_empty_string=ignore_empty_string, ignore_unknown_columns=ignore_unknown_columns),
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2295, in buildQuery
    result = self.buildSingleQuery(key, value)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2087, in buildSingleQuery
    search_key, related_key_definition = self.getColumnSearchKey(key, search_key_name)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2049, in getColumnSearchKey
    related_key_definition = self.getRelatedKeyDefinition(key)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1999, in getRelatedKeyDefinition
    for entire_definition in self.getSQLCatalogRelatedKeyList([key]):
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1935, in getSQLCatalogRelatedKeyList
    column_map = self._getSQLCatalogRelatedKeySet()
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 130, in wrapper
    result = transactional_cache[cache_id] = method(wrapped_self)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1908, in _getSQLCatalogRelatedKeySet
    column_map = self.getColumnMap()
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 130, in wrapper
    result = transactional_cache[cache_id] = method(wrapped_self)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1146, in getColumnMap
    for field in table_dict[table]:
KeyError: 'user'
```

This means that even if upgrader's post upgrade constraint are supposed to fix this by calling `portal_catalog.upgradeSchema`, because this constraint rely on alarm tool, it's already too late.

The suggested way to fix this is to also call [`portal_catalog.upgradeSchema`](https://lab.nexedi.com/nexedi/erp5/blob/c99fb9163503c5afdef59ecb124b3060b05330a4/bt5/erp5_upgrader/SkinTemplateItem/portal_skins/erp5_upgrader/TemplateTool_checkTableConsistency.py#L7) in the same transaction that the transaction upgrading business templates.

I have not completely removed `TemplateToolTableConsistencyConstraint`, because maybe there will be cases where just call post upgrade alarm to execute data migration steps after installing a business template manually. But you guys think it's better to completely merge `TemplateToolTableConsistencyConstraint` with `TemplateToolBusinessTemplateInstallationConstraint` I would be OK with that too.

Of course, we cannot just make
`TemplateToolTableConsistencyConstraint` an `upgrade` constraint, because there would be no guarantee that it's called after `TemplateToolBusinessTemplateInstallationConstraint`.

Also, I have not considered making [`getColumnMap` and friends](https://lab.nexedi.com/nexedi/erp5/blob/master/product/ZSQLCatalog/SQLCatalog.py#L1142) resilient to the case where a table listed in search tables does not exist, because it's would just have been failing later anyway and it's better to fail early in such case.

cc: @vpelletier @jm @seb @tiwariayush @gabriel

/reviewed-on nexedi/erp5!247

We had an incident in a in instance were a user changed state of 70K invoices using https://www.erp5.com/howto/erp5-developer-howto/erp5-HowTo.Change.Workflow.State.Of.Multiple.Documents  and leads to user not receiving the reports they requested.

`Folder_modifyWorkflowStatus` creates a lot of `callMethodOnObjectList` activities (for all the selected documents) with priority 2, then these activities will cause more reindex an expand activities.

Until all these `callMethodOnObjectList` are processed, no new activities with priority > 2 were processed. Some "important for users" activities such as erp5_deferred_style reports where waiting in the queue.

I believe we should just set a lower priority to these `callMethodOnObjectList`, eventhough I considered a more clever way of giving the priority:

 - if the number of selected documents is reasonably small, process them with a very high priority, this way the user can see his document changing state almost immediately as when using the synchronous change state. This should not cause congestion because there are not too many documents.
 - when there are a lot of selected documents, process them with a very low priority, because anyway it will take time and user will not "wait" for documents to change state.

... but I realized this is trying to be too clever.

So any objections to just lower priority here ?

/reviewed-on nexedi/erp5!235

As discussed in  nexedi/erp5!248 this approach allows to set any mail header.

I also included a not so related patch of email header handlings 88d40b40 so that we review all this together.

/cc @gabriel @kazuhiko 

/reviewed-on nexedi/erp5!256

I just simulated an upgrade for an ERP5 running a version from before !185,  `ERP5UserManager Non Existence_constraint` properly disabled old `acl_users/erp5_users` and migrated all persons, but  `ERP5UserLoginManager Existence_constraint` did not activate `acl_users/erp5_login_users`.

This is because  `ERP5UserLoginManager Existence_constraint` is registered as a **pre-upgrade** constraint, but with my upgrade scenario, pre-upgrade constraint are installed too late. 

The upgrade scenario was:

 1. install new ERP5 SR on slapos
 2. manually install new versions of  erp5_upgrader and customer_configuration_upgrader business template
 3. run upgrade ( sense then fix on portal_alarms/promise_check_upgrade )

During step 3, when pre-upgrade constraints are executed, erp5_base is not installed yet, so  `ERP5UserLoginManager Existence_constraint` can not run because it's not installed yet. 

My suggestion is to change it to a post-upgrade constraint.

If I understand correctly and my upgrade scenario is really the supported one, it means we can only have pre-upgrade constraints in erp5_upgrader.

( I also fix a meaningless typo in the same code)

/cc @kazuhiko  @vpelletier @seb @gabriel 

/reviewed-on nexedi/erp5!262

To be able to use pre_upgrade we would need to install the bt first

As Pay Sheet Lines, Pay Sheet Transaction Lines and Annotation Lines
have been made generic using the Invoice Line class

As Pay Sheet Line has now Invoice Line as base class

Because of ERP5Type.UnrestrictedMethod, 'System Processes' can own objects.
Such objects can be proxy-role'd scripts, and proxy-role mechanism
triggers many users look-ups (each time security is evaluated, which is
virtually every getattr). Each such lookup will do a query for 'System
Processes' user, which will (hopefully) find nothing anyway.
So special-case 'System Processes' when looking by user_id by skipping
the search altogether (enforcing the inability to locate this user,
consistently with Zope assumptions, and consistently with previous
behaviour).

The code was not working as expected when we had several test nodes with same name
(which should not happen). So add a timestamp in the title.

test_result: fixed random issues where test result was staying confirmed while all tests were finished

When two test results line were stopped in parallel, both transactions were thinking that
there was still one ongoing test line, thus none of them were setting the test result in stopped.

Use activities instead of serialization to avoid having testnode getting conflict errors.

This makes a difference for SQLDict as not all messages are accepted for
insertion.

Firefox and Xvfb aren not provided by the tested SR,
but rather by a separate SR that is installed by the testnode.

Charting libraries are better within iframe to avoid enabling too permissive
Content Security Policy globally

Fix error display after save
Fix parameters not recognized
Fix python linting
the same way as ERP5 Script Python

Add PortalType field to distinguish different types of Python Scripts, add referencence and use source code editor