Recap on ZODB conflicts:

• conflicts are what keeps ZODB consistent in face of concurrent changes
• some conflicts can be resolved, when their class implements it
• even resolved conflicts have a cost, which is especially visible with NEO as non-conflicting transactions do not have to wait on each other with NEO (they do have to wait on each other with ZEO)

So, splitting hotspot objects, even always-conflict-resolving ones, can make a significant performance difference, especially with NEO. This is what this patch set is about.

Hot-spot 1: portal_ids in-ZODB generator

portal_ids 's in-ZODB id generator works by storing the latest generated value of each sequence they manage as values in a b-tree. So the Persistent instance being modified is the b-tree leaf, which contains a roughly fixed number of entries (around 10). When there are many different sequences being used at the same time (ex: when lettering accouting transactions for many third-parties, with one lettering sequence per third-party), the b-tree leaf sees conflicts happening (whenever two sequences being used happen to reside in the same leaf block).

The solution is to decentralise such sequence state: we already have Persistent objects in ERP5 representing individual third-parties, so storing the grouping reference generator state on the corresponding third-party document allows spreading changes to completely different objects, removing the conflict.

This is implemented in ERP5Type.Base so it is not limited to lettering or even accounting: each ERP5 document gets its own in-ZODB id sequences.

Chosen persistent structure is:

<ERP5Type.Base instance> -> <OOBTree (and its internal structure)> -> <PersistentContainer (and contained integer)>

As in portal_ids, a b-tree in an attempt to avoid generating conflits (though likely resolvable by OOBTree class) if multiple sequences are created in parallel on the same document.

Then, unlike portal_ids, a minimal persistent container is used to wrap the current generator state. This avoids modifying the b-tree leaf object on each id generation, avoiding the problematic conflict.

Id generation API is a variation from portal_ids':

def generateIdList(self, group, count=1, default=1, onMissing=None, poison=False):

API differences with portal_ids:

• There is no "single id" version. Because what can generate lists can generate a list of 1 element.
• Stricter checks on parameters.
• id_group became group: we are already talking about id generation, so no need to repeat that this is about groups of ids.
• onMissing and poison are needed for on-the-fly migration (the former when migrating to this new id generation scheme, the latter when migrating away from it)

Intended usage when migrating (illustrating onMissing and poison):

# Groups declared next to one another to show the difference
old_group = ('just_an_example', some_category_path, document_path)
new_group = ('just_an_example', some_category_path)
new_id, = document.generateIdList(
  group=str(new_group),
  onMissing=lambda: portal_ids.generateNewIdList(
    id_group=str(old_group),
    id_generator='zodb_continuous_increasing', # Whatever the previous generator was
    poison=True, # Prevent any more ids being generated for the old group on portal_ids
  )[0],
)

With this approach:

• thanks to onMissing, you do not have prevent usage of the involved id sequences until fully migrated. It is still advisable to migrate all in a finite time, otherwise both id generators will be involved whenever an actual new group is used.
• thanks to poison, you know if you forgot to update a piece of code using the same sequence, as it will fail rather than silently cause duplicate values

I implemented both on the new id generator.

I backported poison to portal_ids. I did not backport onMissing.

Intended usage outside of migration:

new_id, = document.generateIdList(
  group=str(('just_an_example', some_category_path)),
)

Hot-spot 2: subdocument counter

BTreeFolder2 and HBTreeFolder2 maintain a counter of the number of document each of their instance contain, as a property on each instance. That property is a hot-spot when multiple process create (or delete, but that is much less frequent) documents in a folder (typically most relevant in modules).

The solution is to split that counter so each zope process modifying a folder get its own subobcject counter. The total number of subobjects of that folder is known by summing all those up. This only happens on containers larger than a pre-determines threshold (1000 in this implementation), with the intent to only affect modules.

Chosen persistent structure is:

<Folder> -> <FragmentedLength with a dict property> -> <Length>

It is expected that adding new zope processes is a rare event (compared to the frequence at which documents are created), and that it is trivially cheap to iterate over as many entries as the number of zope processes the database has seen in its lifetime. Thanks to this, FragmentedLength is rarely modified (as it would be a source of conflicts). If it does get modified, it is able to resolve conflicts in order to not cause a whole-transaction retry even in such rare event.

Leaf objects are the same class as before, to not risk regression when the concurrence happens between threads within a process and not different processes. If this somedays becomes a limitation, it can be resolved by attributing stable identifiers to zope threads and extending the key used in FragmentedLength with that identifier.

Hot-spot 3: ZSQLCatalog.SQLCatalog's copy of latest uid

For completeness: I already pushed a hot-spot removal in ZSQLCatalog, because it was even easier: there was no point in having that object to begin with, even if it was not a conflict hotspot.

/cc @romain @jerome @jm @jp @seb @tb