• Vincent Pelletier's avatar
    Use REPLACE rather than INSERT for DELETE..INSERT indexation pattern. · 5d3ea021
    Vincent Pelletier authored
    When a document gets rows inserted in category table while there was none
    before (typically: first document indexation), it may trigger
      IntegrityError: (1062, "Duplicate entry '...' for key 'PRIMARY'")
    because in the DELETE..INSERT pattern, DELETE finds no matching rows and
    does not gap-lock (because we enable innodb_locks_unsafe_for_binlog), then
    the second INSERT happening, chronologically speaking, waits for the
    transaction of the first to e committed, and on commit it causes such
    duplicate key error.
    
    A transient visible effect of this change is that if both competing
    indexations see a different document state (because document got modified
    in some 3rd transaction), the union of the resulting set of rows will be
    visible until the reindexation which must have been triggered by the 3rd
    transaction gets executed, at which point only the latest set will be
    visible.
    
    A similar issue exists before this change, with stricter conditions: it
    needs the intersection of both sets to be empty, because a non-empty
    intersection causes the duplicate key error solved here.
    
    This change has been measured to improve scalability of an invoice building
    test case (naturally triggering indexations) starting from ~12 activity
    nodes:
       8 nodes:  +1.4% invoices/hour
      12 nodes:  +9.5% invoices/hour
      16 nodes: +12.3% invoices/hour
    (values are the difference between DELETE..INSERT and DELETE..REPLACE)
    5d3ea021
z_catalog_non_movement_category_list.xml 2.93 KB