nexedi / erp5

In such case, it must obviously be considered that the ZODB Component Document
was not found and thus fallback on filesystem Document.

2018-11-21 17:05:01,919 WARNING ERP5Type.Dynamic Could not access Portal Type Object for type 'Category Tool'
Traceback (most recent call last):
  File "product/ERP5Type/dynamic/lazy_class.py", line 3
52, in loadClass
    class_definition = generatePortalTypeClass(site, portal_type)
  File "product/ERP5Type/dynamic/portal_type_class.py",
 line 198, in generatePortalTypeClass
    module = erp5.component.document.find_load_module(type_class)
  File "product/ERP5Type/dynamic/component_package.py",
 line 361, in find_load_module
    loader = self.find_module(fullname)
  File "product/ERP5Type/dynamic/component_package.py",
 line 155, in find_module
    component_tool = aq_base(site.portal_components)
AttributeError: portal_components

/reviewed-on !799

Using a cache cookie that is incremented each time an account is added or modified,
we can keep an (almost) up to date list of accounts in the UI.

Fixes #20170221-1812E21

/reviewed-on !796

Before this change, `index_html` for Catalog Tool was None that means trying to
access view for CatalogTool from UI always make a function call to the
` __call__` which function was overridden with `searchResults`, which
means everytime someone tried to access `portal_catalog/`, a useless query was
being made and an undesirable query result was displyed on the UI.

In this case, `index_html` is prefered whenever we try to access portal_catalog
from UI.

Also, add test for proper display of view when CatalogTool is accessed from XHTML UI

/reviewed-on !793

.pop .popitem and .setdefault where returning None

/reviewed-on !791

Called by TransformationSimulationRule and did not work with the
following use case:

  BL(trade_phase=TP1, successor=trade_state/TS1):
    * child_BL1(predecessor=trade_state/TS1,
                trade_phase=TP1/InventoryAccounting)
    * child_BL2(predecessor=trade_state/TS1,
                trade_phase=Transforming)

in accounting_module and sale_order_module listbox.

The "Sale Trade Condition" column was showing first found specialise title
instead of the Sale Trade Condition one.

This adds z_related_specialise_trade_condition with related key specialise_trade_condition_title
and an accessor getSpecialiseTradeConditionTitle

When a workflow transition was not used by any state,
calling some_module.Module_listWorkflowTransitionItemList was raising:
    [..]
      Module script, line 49, in Module_listWorkflowTransitionItemList
       - <PythonScript at /tsxx/Module_listWorkflowTransitionItemList used for /tsxx/some_module>
       - Line 49
        result['listbox_parameter_dict'][transition_id] = [(state_variable, allowed_state_dict[transition_id])]
      Module AccessControl.ZopeGuards, line 83, in guarded_getitem
        v = object[index]
    KeyError: 'some_action'

Now, pick transition among the allowed_state_dict instead of workflow.transitions.

This also fixes mass change state dialog "??? ()" issue.
This commit adds a transition that allows to reproduce that issue.
And this line covers the case :
bt5/erp5_web_renderjs_ui_test/PathTemplateItem/portal_tests/renderjs_ui_zuite/testMassWorkflowTransition.zpt (line 61)

/reviewed-on !788

Such as:
  - getStartTransitionDateList() and getStartTransitionDate().
  - getDeliverTransitionDateList() and getDeliveryTransitionDate().
  - ...

Where the non-List accessors returned the last element.

Use Case: Set AT.{start,stop}_date from PPL stop transition date.

Until this commit, loading a ZODB Component would add it to the current Request
object to prevent its reference counter reaching 0 (thus its global being reset to
None and then being later GC'ed) when a reset happens in another Request.

This problem was found when investigating testFunctionalConfigurator failure
when installing bt5s as requested by the Configurator:
  1. Request R1 calls Zuite_waitForActivities importing extension.erp5.ERP5Zuite (M1).
     => Add M1 to R1._module_cache_set.
     => M1 ref counter equals to 2 (sys.modules and R1._module_cache_set references).
  2. R1 terminates and is GC'ed.
     => M1 ref counter equals to 1 (sys.modules).
  3. Request R2 runs Configurator configuring the Site.
  4. testFunctionalConfigurator calls Zuite_waitForActivities to wait for the
     Configurator to finish (request R3 which may take ~15 minutes). This calls
     time.sleep() in a loop where 'time' module is imported at top-level.
  5. R2 installs bt5 triggering reset.
     => M1 ref counter equals to 0.
        ===> M1 global variables are reset to None and thus 'time' is set to None
             raising an Exception in the next call of time.sleep() in the loop.

The easiest way would be to have a hook on sys.modules dict lookup and thus add M1
to R2._module_cache_set when being imported, but this is not supported... Instead
create a global cache on erp5.component package.

This test alone does not define site-dependent security groups, so this
change is not required as such.
But in real configuration, ERP5 instance define security groups, which
sometimes (often ?) depend on which site a document or user is linked to.
When multiple groups apply (ex: having a certain function at a certain
site), the final group is composed by joining group identifiers using
underscores. This forbids using underscores in group identifiers.
And group identifiers are generated from document's (here, the Category)
codification if set, otherwise the reference if set, otherwise the document
identifier.
So set a underscore-less codification on these categories so this test can
be run on tests where security is configured to include site in security
groups.

as the comment (which is removed now) said:

```
        XXX: This method is broken, since "total_size" field is not
        present in the listbox rendering any longer. It should be
        removed.
```

Also:
- make sure that Listbox always knows how many lines are to be displayed.
- add testPagerWithCustomListMethod in listbox_zuite of erp5_ui_test,
which checks that listbox jumping to previous/next/first/last works correctly
for a list method that returns only the elements that the listbox is to render.


/reviewed-on !776

set.__iadd__ does not like non-set values, but the intent of this API is
to allow just any iterable (producing strings) to be returned, so use
the more tolerant set.update method instead.

This allows breaking a circular dependency of catalog on itself: to
index a local role, catalog must tell whether it's granted to a group or
to a user, which used to happen using a catalog lookup (via PAS API),
preventing catalog from being rebuilt from scratch.

With this change, ERP5Site_filterUserIdSet must be implemented when
custom role columns are used (viewable_owner is not concerned by this
change).
If it is implemented in a way which does not rely on catalog,
catalog become rebuildable from a lone ZODB, without double-indexation
of users nor security_uid explosion.
Nothing prevents relying on catalog (which mayu be desired for
backward-compatibility purposes), but such script will not be provided
by generic ERP5, as the impossibility of recreating catalog from ZODB
is considered a bug, and introducing such script amounts to introducing
this bug.

Create ERP5Site_filterUserIdList in tests which add custom viewable_*
columns.

Use Owner role to help filling user_set: owners must be users, never
groups.
Only look for users among group_ids when they are candidate for indexation
into a catalog_role column. This should very significantly reduce the
number of user lookups, even bringing it to zero on instance with default
catalog setting of catalog_role_set = set(["viewable_owner"]) .
Also, only look for users if there are candidates for lookup.
Also, rename local variable to better describe its content.
Also, use set.__iadd__ instead of set.update (shorter).

Test loop invariant outside of loop.
Use clearer and more consistent variable names.
Avoid iterating twice when once is enough, allowing removal of
"local_role_dict" local and simplification of corresponding dict
structure.
Avoid creating new instances when the existing one can be mutated.
Evaluate dict as boolean instead of constructing an empty dict to compare
against.
Avoid using %-operator just to concatenate strings (allows en-passant
typechecking).
Drop unused locals.
Do not use a dict when a set would be sufficient.

So that it can be reused outside of group manager without having to
duplicate the backward-compatibility order.
Also, simplify code.

Move variable initialisations closer to where they are actually used.
Do not initialise a variable which will always be initilised again before
its first use.
Reduce argument indentation.
Drop unhelpful comments.
User document may not be a Person instance, use a more neutral variable
name.
Avoid single-use locals.
Take advantage of zLOG.LOG's "error=True" idiom to not have to pull
exception information ourselves.

Fixup bf57228a

This reduces the number of logs.

When editing components, this warning is displayed on the console:
```
Ambiguous name for method of Products.ERP5.Document.SQLMethod.SQLMethod: 'manage_main' != 'manage'
```
fix by adding the same `_setName` that the [parent class does](https://github.com/zopefoundation/Products.ZSQLMethods/blob/2.13/src/Products/ZSQLMethods/SQL.py#L143).


While looking a this, I realized that clicking on the breadcrumb in the ZMI, any part of

![image](/uploads/5e42659d8124b7566fb09cc76958a345/image.png)

would redirect to the ERP5 xhtml style view.

/reviewed-on !782

It is extremely slow for some reason on large BTreeFolders inside
BTreeFolder2's has_key (even on warm caches). Using get instead is
significantly faster.
Found by chance, calling getAcquiredCategoryList on a random Person when
person_module contains of the order of magnitude of a million documents.

we used to add this value because ```select``` without ```option```
(that happens in MultiListField only) is invalid on XHTML.

But it was not at all required for other cases like MultiCheckboxField or RadioField.

Also,  ```select``` without ```option``` is no longer invalid on HTML5.

Considering that showing dummy unselectable <option> is anyway meaningless for users,
we drop it for all cases.

/reviewed-on !777

The warning is:
```
Ambiguous name for method of Products.ERP5.Document.SQLMethod.SQLMethod: 'manage_main' != 'manage'
```

It happens when components are reloaded

This is needed when running livetests in a runUnitTest instance, in the
following scenario:
 - `runUnitTest --save test_bt:testXXX`
 - `runUnitTest --save --load`
 - run the live test from the browser

because livetest way of processing activities is to unsubscribe at the
beginning of the test and subscribe at the end to have control of
activities when running .tic, if ProcessingNodeTestCase also process the
activities automatically, they interfere and test often fail with errors like:

```
File "ERP5Type/tests/ProcessingNodeTestCase.py", line 249, in tic
  raise RuntimeError(error_message)
RuntimeError: tic is looping forever. These messages are pending: [('/erp5/portal_components/test.erp5.testSupportRequest', 'immediateReindexObject', 1, 0), ('/erp5/support_request_module/2617', 'immediateReindexObject', -1, 0)]
```

/reviewed-on !779

89115b88 is great, but with "testFunctionalAnonymousSelection" the
result HTML is more than 40Mo.

/reviewed-on !772

This will insure that the URL created for diff by `Base_redirect` will have the paramters
`your_first_path` and `your_second_path` in it, thus making it unique and hence
sharable. Before, we only used to have the name of selection in the URL which
returned different result for different users.

This was inefficient for two reasons:
- any message we could validate during current iteration means a message we
  did not consider is now in the range we just scanned. And it will not be
  considered until validation node starts over and scan this same range
  again.
- "LIMIT x,1000" pattern on >1000 messages causes a quick-growing number of
  extra rows scanned by the SQL database just to skip the "x" first rows:
  at 2000 rows present it must scan 1000 + 2000 = 3000 rows for a complete
  loop over all pending activities. At 3k rows it must scan 6k rows.
  At 4k, 10k.
  While this is an overestimation (some rows should be possible to
  validate, so these would be scanned once only), this overhead grows so
  large that this overestimation can become negligible.

Instead, use a range condition consistent with query's "SORT ON", which is
already efficiently materialised by an index: SQL database just has to
dive into the existing index to start just above the last message from
previous iteration, and resume scanning from there, solving both issues
listed above.

This method is called on every __getattr__ (guarded_getattr, actually) of
every restricted python scripts. Which means each "." in the code triggers:
- traversal to relevant acl_user folder
- a user lookup
just to get the same value throughout the execution of a script (as its
owner does not change during execution).
Also, in practice we have extremely few possible owners: very few users are
allowed to edit code in an ERP5 instance, and if such instance is managed
using the upgrader alarms, they will even further reduce this owner set to
System Processes only.
On a real-world web page rendering, this reduces the total number of
traversal calls from 1500 to 1100, getting rid of the two hottest spots:
/acl_users and /$site_id/acl_users .