1. 01 Mar, 2019 1 commit
    • Georgios Dagkakis's avatar
      erp5_core: Ignore non-existent transitions in Module_listWorkflowTransitionItemList · 14940e3f
      Georgios Dagkakis authored
      instead of crashing.
      
      This can happen in workflow if we have:
      - transition_x is declared as possible in state_x
      - then transition_x gets deleted
      In this case in state_x in manage_properties nothing is visible,
      which creates an inconsistent in its data, yet functional generally workflow
      
      Since it is easy to create such cases, just ignore them in Module_listWorkflowTransitionItemList
      so that mass change state can work
      14940e3f
  2. 27 Feb, 2019 1 commit
  3. 26 Feb, 2019 3 commits
  4. 25 Feb, 2019 1 commit
  5. 21 Feb, 2019 3 commits
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      CMFActivity: new activate() parameter to prefer executing on the same node · 301962ad
      Julien Muchembled authored
      The goal is to make better use of the ZODB Storage cache. It is common to do
      processing on a data set in several sequential transactions: in such case, by
      continuing execution of these messages on the same node, data is loaded from
      ZODB only once. Without this, and if there are many other messages to process,
      processing always continue on a random node, causing much more load from ZODB.
      
      To prevent nodes from having too much work to do, or too little compared to
      other nodes, this new parameter is only a hint for CMFActivity. It remains
      possible for a node to execute a message that was intended for another node.
      
      Before this commit, a processing node selects the first message(s) according to
      the following ordering:
      
        priority, date
      
      and now:
      
        priority, node_preference, date
      
      where node_preference is:
      
        -1 -> same node
         0 -> no preferred node
         1 -> another node
      
      The implementation is tricky for 2 reasons:
      - MariaDB can't order this way in a single simple query, so we have 1
        subquery for each case, potentially getting 3 times the wanted maximum of
        messages, then order/filter on the resulting union.
      - MariaDB also can't filter efficiently messages for other nodes, so the 3rd
        subquery returns messages for any node, potentially duplicating results from
        the first 2 subqueries. This works because they'll be ordered last.
        Unfortunately, this requires extra indices.
      
      In any case, message reservation must be very efficient, or MariaDB deadlocks
      quickly happen, and locking an activity table during reservation reduces
      parallelism too much.
      
      In addition to better cache efficiency, this new feature can be used as a
      workaround for a bug affecting serialiation_tag, causing IntegrityError when
      reindexing many new objects. If you have 2 recursive reindexations for both a
      document and one of its lines, and if you have so many messages than grouping
      is split between these 2 messages, then you end up with 2 nodes indexing the
      same line in parallel: for some tables, the pattern DELETE+INSERT conflicts
      since InnoDB does not take any lock when deleting a non-existent row.
      
      If you have many activities creating such documents, you can combine with
      grouping and appropriate priority to make sure that such pair of messages won't
      be executed on different nodes, except maybe at the end (when there's no
      document to create anymore; then activity reexecution may be enough).
      For example:
      
        from Products.CMFActivity.ActivityTool import getCurrentNode
        portal.setPlacelessDefaultReindexParameters(
          activate_kw={'node': 'same', 'priority': priority},
          group_id=getCurrentNode())
      
      where `priority` is the same as the activity containing the above code, which
      can also use grouping without increasing the probability of IntegrityError.
      301962ad
    • Nicolas Wavrant's avatar
      8fa3d858
  6. 20 Feb, 2019 2 commits
  7. 18 Feb, 2019 3 commits
  8. 16 Feb, 2019 1 commit
  9. 15 Feb, 2019 2 commits
  10. 13 Feb, 2019 5 commits
  11. 12 Feb, 2019 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components: Likewise Document, add Mixin (erp5.component.mixin) and... · e84d2b51
      Arnaud Fontaine authored
      ZODB Components: Likewise Document, add Mixin (erp5.component.mixin) and Interface (erp5.component.interface).
      
      * One Mixin/Interface class per ZODB Component.
        => Already the case for FS Mixin, not for Interfaces.
      * ZODB Components module name ('reference' property) and class name:
        + Mixin: FooMixin.
        + Interface: IFoo.
      
      Rationale:
        + Avoid current FS hacks: registry (Mixins, mixin_class_registry) or import
          all classes explicitly in __init__.py (Products.ERP5Type.interfaces).
        + Consistent naming.
        + Consistent with ZODB Documents Components.
      
      Also, modify pylint checker to handle Zope Interfaces:
        + E: 4, 0: Inheriting 'Interface', which is not a class. (inherit-non-class)
        + E: 5, 2: Method has no argument (no-method-argument)
      e84d2b51
  12. 07 Feb, 2019 1 commit
  13. 05 Feb, 2019 8 commits
    • Julien Muchembled's avatar
      4b7acaa7
    • Julien Muchembled's avatar
      CMFActivity: remove old skin if any · afaa9d19
      Julien Muchembled authored
      afaa9d19
    • Julien Muchembled's avatar
      CMFActivity: drop DTML completely and use consecutive uids when possible · d64887cb
      Julien Muchembled authored
      This moves the remaining DTML queries to Python, dropping the 'activity' skin.
      
      Dealing with conflicts of uids is easier if the inserted uids are consecutive:
      now, only 1 random value is generated, as base uid. This also preserves the
      order of insertion, which is wanted for performance reasons:
      - No more random write in the primary index.
      - When modifying several lines of several documents, 1 document being processed
        at a time, we'd like that any grouped activity (usually indexation) follows
        the same order, so that a processing node prefer many lines from a few
        documents instead of mixing lines from too many documents at the same time.
        This is usually better for caches.
      d64887cb
    • Julien Muchembled's avatar
      CMFActivity: speed up console activity watcher · bcaf44a4
      Julien Muchembled authored
      Average age of activities is dropped because it would become
      too complicated to implement and it's useless information.
      bcaf44a4
    • Julien Muchembled's avatar
      CMFActivity: remove processing/processing_date columns and improve watchers · 499d8f8b
      Julien Muchembled authored
      The original goal was to improve performance by removing the
      `processing_node_processing` index and the queries that modified
      these 2 useless columns.
      499d8f8b
    • Julien Muchembled's avatar
      CMFActivity: do not refresh view too often when checking messages dependencies · a32c8a42
      Julien Muchembled authored
      The root call to getExecutableMessageList (i.e. the one from distribute)
      is fast enough and won't hold old revisions of the database for too long.
      It is also completely read-only so it won't lock anything.
      
      This caused useless communication with the server.
      a32c8a42
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      CMFActivity: validate a message with a single SQL request per queue · 7e387bcb
      Julien Muchembled authored
      As shown in the following example, on a big catalog table,
      MariaDB is able to use several indices at the same time
      ('...' are obfuscated unique values):
      
        > analyze select SQL_NO_CACHE uid, relative_url from catalog where reference='...' OR relative_url='...';
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        | id   | select_type | table   | type        | possible_keys          | key                    | key_len | ref  | rows | r_rows | filtered | r_filtered | Extra                                            |
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        |    1 | SIMPLE      | catalog | index_merge | Reference,relative_url | Reference,relative_url | 768,767 | NULL |    2 |   2.00 |   100.00 |     100.00 | Using union(Reference,relative_url); Using where |
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        1 row in set (0.00 sec)
      
      So mixing different dependency types with OR should be fine
      (no need to split into more subqueries and join with UNION).
      7e387bcb
  14. 31 Jan, 2019 1 commit
  15. 29 Jan, 2019 2 commits
  16. 21 Jan, 2019 1 commit
  17. 18 Jan, 2019 4 commits