1. 15 Sep, 2022 2 commits
  2. 14 Sep, 2022 1 commit
    • Vincent Pelletier's avatar
      Products.CMFActivity.Activity.SQLBase: Drop "FOR UPDATE". · 89aa2a6b
      Vincent Pelletier authored
      This SELECT option was added when the UPDATE query was split into a
      SELECT + UPDATE query pair, in order to try to avoid deadlocks in mariadb.
        commit 7daaf0a5
        Author: Vincent Pelletier <vincent@nexedi.com>
        Date:   Fri Jun 7 16:53:11 2013 +0200
      
            Work around poor UPDATE use of index.
      That was not completely successful, and another attempt was since made,
      which further ipmproved the situation:
        commit e21aa47f601a98f767e2d3e8490b20b7f409933c
        Author: Vincent Pelletier <vincent@nexedi.com>
        Date:   Fri Sep 17 11:34:46 2021 +0900
      
            CMFActivity.Activity.SQLBase: Reduce the number of deadlocks
      
      This change made the "FOR UPDATE" option obsolete: there should be
      virtually no parallelism between the SELECT and UPDATE queries.
      By which I mean only 3rd-party accesses (like an admin on a mysql console)
      would bypass the SQL lock, causing index-update-level locking issues.
      
      Checking now, I see that removing "FOR UPDATE" on the "node>0" query makes
      that query (with LIMIT 30) run about 5 times faster:
      With FOR UPDATE:    0.509s
      Without FOR UPDATE: 0.090s
      This difference is enough to cause significant activity starvation on a
      63-processing-nodes cluster where such activities run in about 1s, just
      because of the amount of contention on the SQL lock, itself because of the
      duration of every execution of this query.
      
      So remove this now-redundant option.
      
      It is expected that this will improve activity processing performance in
      general, beyond just the starvation issue which led to this discovery.
      89aa2a6b
  3. 09 Sep, 2022 1 commit
  4. 07 Sep, 2022 3 commits
  5. 06 Sep, 2022 2 commits
    • Kazuhiko Shiozaki's avatar
      Folder: fix objectValues(portal_type=...). · 48cfa07a
      Kazuhiko Shiozaki authored
      For example, Web Section can contain a Script (Python) to set Access
      Rule, but the parent section's objectValues(portal_type='Web Section')
      should not include such object.
      48cfa07a
    • Vincent Pelletier's avatar
      Products.CMFActivity: Fix poor performance with many family-bound activities · 62af8254
      Vincent Pelletier authored
      When there are many simultaneously-pending activities attached to any
      processing node family, the
        node>=0
      subquery becomes dominant (taking hundreds of time longer than the other
      subqueries). As a consequence, this starves processing nodes of activities
      and increases the CPU needs of the mariadb process hosting the activity
      tables.
      So, move this subquery out of the regular codepath, and only run it if no
      other subquery found any activity:
      - there is no activity preferentially targeting the current node
      - there is no activity bound to any of the current node's families
      - there is no activity without any node preference at all
      Also, simplify the content of that subquery: the effective priority can
      only be 3 * priority + 1 when this query is run, and node=0 rows can be
      excluded (they should not exist in the current database view).
      Also, factorise the logic producing "node=processing_node" and
      "node IN node_set" subqueries, for simplicity. In turn, this makes all
      family-dependent subqueries use a simple equality test, ensuring a stable
      query plan independently from the number of families the current node is
      member of.
      Also, use "UNION ALL" always, as now:
      - all subqueries have stritly distinct result sets
      - as per mariadb documentation, "UNION [DISTINCT] applies to all UNIONs
        on the left", so the original comment about where ALL is used was
        incorrect in assuming it was improving the effective query performance
      Also, line-split SQL queries as visible in the python source to be more
      readable, without effect on the produced SQL.
      Also, line-split a few non-trivial python expression to make their
      internal structure immediately apparent.
      
      Another effect of this change this change is to reduce activity theft
      (activities to be preferentially executed by one node being executed by
      another), potentially improving object cache hit-rate and hence decreasing
      I/O pressure on the ZODB.
      62af8254
  6. 30 Aug, 2022 1 commit
  7. 24 Aug, 2022 1 commit
  8. 22 Aug, 2022 4 commits
    • Jérome Perrin's avatar
      xhtml_style: remove hashes from icons.png URLs · 1b4d4ba9
      Jérome Perrin authored
      this change was made with:
      
          find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/\\?t=445cf24ebd//g'
      1b4d4ba9
    • Jérome Perrin's avatar
      xhtml_style: remove timestamp from ckeditor URLs · 6e8cc3c9
      Jérome Perrin authored
      This timestamp break officejs offline capabilities.
      
      This reaplies 40f6c8fe ([erp5_xhtml_style] ckeditor: drop hardcoded
      timestamp parameter in URL, 2017-10-16) on the updated CKEditor
      
      This was done with:
      
          find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/"M6K9"/""/g'
      
      M6K9 was the timestamp for this build, that we can see at the top of
      ckeditor.js
      6e8cc3c9
    • Jérome Perrin's avatar
      c3f769b4
    • Jérome Perrin's avatar
      xhtml_style: version up ckeditor 4.19.1 · 5e82e22a
      Jérome Perrin authored
      done with an external method to upload the content of zip file in
      skin folder.
      
          import zipfile
          from io import BytesIO
      
          def uploadZip(self):
            dest = self.getPortalObject().portal_skins.custom
            with zipfile.ZipFile(
                '/srv/slapgrid/slappart3/srv/project/erp5/ckeditor_4.19.1_4b98d281bc34.zip'
            ) as f:
              for m in f.namelist():
                current_dest = dest
                print(m)
                for part in m.split('/')[:-1]:
                  if part not in current_dest.objectIds():
                    current_dest.manage_addProduct['OFSP'].manage_addFolder(id=part)
                  current_dest = current_dest[part]
                fname = m.split('/')[-1]
                if fname == '_translationstatus.txt':
                  fname = 'translationstatus.txt'
                current_dest.manage_addProduct['OFSP'].manage_addFile(
                  id=fname, file=BytesIO(f.read(m)))
            return repr(dest)
      5e82e22a
  9. 18 Aug, 2022 6 commits
    • Jérome Perrin's avatar
      Listbox,SelectionTool: use make_query instead of crafting URL manually · 3cb786cc
      Jérome Perrin authored
      Using make_query makes sure that the query parameters are properly
      encoded and also change selection_index to become an int, because
      it keeps the type of parameters.
      
      As a consequence, we had to adjust a few place in the code where
      selection_index was tested for truthiness: because "0" as a string is
      true, but 0 as an int is not. For that, we changed to test the presence
      of selection_name instead of testing selection_index, as they are always
      used together.
      
      This fixes a problem that &amp; in URL was encoded twice for listbox
      anchor links ( bug_module/1137 )
      3cb786cc
    • Jérome Perrin's avatar
      *: reorganise indexation methods · 2bd1d4ed
      Jérome Perrin authored
      02011d8e (immediateReindexObject: use super user to reindex script,
      2015-12-14) did not apply for inventory, because they were overloading
      immediateReindexObject.
      Introduce a new level, _immediateReindexObject that will hold the actual
      reindexing logic.
      
      Previously this method was using PortalContent.reindexObject which was
      monkey patched, to make things less complex and more future proof, move
      the monkey patch to a method on base class.
      
      This also drops alternateReindexObject on BalanceTransaction, because it
      is already defined in Inventory
      2bd1d4ed
    • Jérome Perrin's avatar
      ERP5TypeLiveTestCase: close request at the end of each test · e605544e
      Jérome Perrin authored
      This is supposed to fix "Should not load state for ${oid of a skin} when
      the connection is closed" sometimes happening with live tests (especially
      when the test self.publish and the developer access the site while the
      test is suspended on a debugger breakpoint).
      
      The object accessed after the connection is closed was a skin (python
      script, sometimes page template or form) that was cached in SKINDATA.
      The mechanism to prune entries from the cache uses REQUEST.hold API
      which expects that REQUEST.close is called on request, but because
      requests were not closed at the end of the request, it happened that
      the cache was reused from another connection.
      
      This change to close the requests, like ERP5TypeTestCase is doing in
      tearDown (the actual close is done by Testing.ZopeTestCase.connections).
      
      By closing requests at the end of tests, we also have to change so that
      at the beginning of the test we initialize the request, by using the same
      setSite and setupCurrentSkin that are done in ERP5TypeTestCase.
      e605544e
    • Jérome Perrin's avatar
      ProxyField: use Skinnable API to get current skin · 8a8f9384
      Jérome Perrin authored
      SKINDATA is an implementation detail that should not be accessed from
      this level.
      8a8f9384
    • Jérome Perrin's avatar
      BusinessTemplate: drop useless fixZSQLMethod in SkinTemplateItem.install · 45e45842
      Jérome Perrin authored
      In SkinTemplateItem.install, self._objects contains entries for skin
      folders and for all skins. objectValues method calls was called for skin
      folders (as expected) and also for all skins, which acquire objectValues
      from skin folder and do the work again.
      
      This simplifies this by only running this for skin folders.
      45e45842
    • Jérome Perrin's avatar
      Listbox: refactor computation of "default" URL · 9ca9d73d
      Jérome Perrin authored
      "default" URL is the URL when listbox column does not use URL column
      and when the brains do not have a getListItemUrl method.
      
      Move the computation in a lazyMethod, so that it is computed only once
      per line instead of once per cell.
      
      Also remove a try/except, I don't think this code is supposed to get an
      AttributeError
      9ca9d73d
  10. 09 Aug, 2022 2 commits
  11. 02 Aug, 2022 10 commits
  12. 28 Jul, 2022 1 commit
  13. 21 Jul, 2022 1 commit
  14. 13 Jul, 2022 1 commit
    • Jérome Perrin's avatar
      accounting: fix grouping dialog loosing section category on next page · 0d8242de
      Jérome Perrin authored
      In xhtml_style, just after clicking next or previous page in the listbox
      buttons, the values selected in 'Section Category' and 'Section Category
      Strict' were not used, because they were used from request and not from
      as normal script parameters.
      
      This fixes only the xhtml_style version, ERP5JS has another problem that
      dialog fields values are reset when going to next page, so it would also
      need this problem to be fixed.
      0d8242de
  15. 12 Jul, 2022 1 commit
  16. 07 Jul, 2022 2 commits
  17. 01 Jul, 2022 1 commit