1. 21 May, 2015 1 commit
  2. 19 Jun, 2014 1 commit
    • Julien Muchembled's avatar
      client: in iterator records, export data serial as stored by NEO · e76af297
      Julien Muchembled authored
      There is simply no way to guess data serials and instead of producing random
      values, the only solution is to retrieve the values from storages.
      
      There are still differences for data serials between FileStorage and NEO:
      - NEO always resolves to original serial, to avoid any indirection
        (which slightly speeds up undo at the expense of a more complex pack code)
      - NEO does not make any difference between object deletion and creation undone
        (data serial always null in storage)
      It has to be decided whether NEO implementation should be changed about this.
      
      Apart from that, conversion database back from NEO should be fixed.
      testExportFileStorageBug passes and there was in fact no FileStorage bug.
      
      Another change is that iterator does not trash the client cache anymore.
      e76af297
  3. 04 Jun, 2014 1 commit
  4. 03 Jun, 2014 1 commit
  5. 07 Jan, 2014 4 commits
  6. 04 Jan, 2014 1 commit
  7. 13 Dec, 2013 1 commit
  8. 26 Aug, 2012 1 commit
  9. 23 Aug, 2012 2 commits
  10. 20 Aug, 2012 1 commit
  11. 23 Jul, 2012 1 commit
    • Julien Muchembled's avatar
      client: really process all invalidations in poll thread · c277ed20
      Julien Muchembled authored
      This changes completely how to get data from storages than is not too
      recent and NEO now behaves as expected by ZODB, instead of trying to
      snapshot at Storage level.
      However, ZODB should probably be changed to avoid double loading when
      an invalidation is received during a load.
      c277ed20
  12. 19 Jul, 2012 1 commit
    • Julien Muchembled's avatar
      client: fix cache invalidation during a load from a storage for the same oid · 04ab2a4c
      Julien Muchembled authored
      This fixes the following an invalidation bug:
      
      ERROR ZODB.Connection Couldn't load state for 0x504e
      Traceback (most recent call last):
        File "ZODB/Connection.py", line 851, in setstate
          self._setstate(obj)
        File "ZODB/Connection.py", line 916, in _setstate
          self._load_before_or_conflict(obj)
        File "ZODB/Connection.py", line 931, in _load_before_or_conflict
          if not self._setstate_noncurrent(obj):
        File "ZODB/Connection.py", line 954, in _setstate_noncurrent
          assert end is not None
      AssertionError
      04ab2a4c
  13. 18 Jul, 2012 1 commit
    • Julien Muchembled's avatar
      client: always process invalidations in poll thread · bce3bc78
      Julien Muchembled authored
      This fixes an invalidation bug, including the following critical error:
      
      CRITICAL txn.140440071526144 A storage error occurred during the second phase of the two-phase commit.  Resources may be in an inconsistent state.
      ------
      ERROR Zope.SiteErrorLog 1342544345.990.582646288246 /erp5/person_module/Folder_create
      Traceback (innermost last):
        Module ZPublisher.Publish, line 137, in publish
        Module Zope2.App.startup, line 291, in commit
        Module transaction._manager, line 93, in commit
        Module transaction._transaction, line 322, in commit
        Module transaction._transaction, line 424, in _commitResources
        Module neo.client, line 42, in tpc_finish
        Module neo.client.Storage, line 135, in tpc_finish
        Module neo.client.app, line 773, in tpc_finish
        Module neo.client, line 36, in callback
        Module ZODB.DB, line 693, in invalidate
        Module ZODB.DB, line 532, in _connectionMap
        Module ZODB.DB, line 221, in map
        Module transaction.weakset, line 58, in map
        Module ZODB.DB, line 692, in inval
        Module ZODB.Connection, line 350, in invalidate
      AssertionError: invalidations out of order, '\x03\x97\xec;\x19\x86\xc9\xf6' < '\x03\x97\xec;\x19\x87_\xdd'
      bce3bc78
  14. 13 Jul, 2012 1 commit
  15. 26 Mar, 2012 1 commit
  16. 21 Mar, 2012 1 commit
  17. 20 Mar, 2012 1 commit
    • Julien Muchembled's avatar
      Review logging to keep all debugging information in RAM and flush only if useful · 1fce5cc4
      Julien Muchembled authored
      The main goal of this patch is to keep all DEBUG logs and packet logger enabled
      without exploding disk usage.
      This is done by keeping the last 16 MB (by default) of debugging information in
      a RAM buffer, and to emit it to an SQLite DB upon RTMIN signal or in case of
      warning and more severe log.
      
      Implementation is also cleaned up for better integration within a framework
      or if run standalone. NEO logger is now a direct child of root handler.
      Only warnings and more severe logs are forwarded to root handler.
      
      A new script 'neolog' is added to pretty-print the contents of the SQLite log.
      
      In unit tests, logging events are not buffered but emitted immediately.
      When a test passes, payloads of all exchanged packets are discarded to reduce
      disk usage on test bots.
      
      This slows down performance tests by about 15 % because even if nothing is
      written to disk, debug and packet log records are now always rendered.
      1fce5cc4
  18. 19 Mar, 2012 4 commits
  19. 16 Mar, 2012 1 commit
  20. 13 Mar, 2012 1 commit
  21. 10 Feb, 2012 1 commit
  22. 19 Jan, 2012 2 commits
  23. 17 Jan, 2012 1 commit
  24. 10 Jan, 2012 1 commit
    • Julien Muchembled's avatar
      client: many fixes to 'transactionLog' · 6aa372d9
      Julien Muchembled authored
      - Do not fetch data from outdated/discarded cells.
      - Do not return more than transactions than requested by 'limit' parameter.
        Anyway, all results above this 'limit' could contain holes.
      6aa372d9
  25. 09 Jan, 2012 1 commit
  26. 06 Jan, 2012 1 commit
  27. 02 Jan, 2012 1 commit
  28. 26 Oct, 2011 1 commit
  29. 11 Oct, 2011 1 commit
    • Julien Muchembled's avatar
      Fix protocol and DB schema so that storages can handle transactions of any size · d5c469be
      Julien Muchembled authored
      - Change protocol to use SHA1 for all checksums:
        - Use SHA1 instead of CRC32 for data checksums.
        - Use SHA1 instead of MD5 for replication.
      
      - Change DatabaseManager API so that backends can store raw data separately from
        object metadata:
        - When processing AskStoreObject, call the backend to store the data
          immediately, instead of keeping it in RAM or in the temporary object table.
          Data is then referenced only by its checksum.
          Without such change, the storage could fail to store the transaction due to
          lack of RAM, or it could make tpc_finish step very slow.
        - Backends have to store data in a separate space, and remove entries as soon
          as they get unreferenced. So they must have an index of checksums in object
          metadata space. A new '_uncommitted_data' backend attribute keeps references
          of uncommitted data.
        - New methods: _pruneData, _storeData, storeData, unlockData
        - MySQL: change vertical partitioning of 'obj' by having data in a separate
          'data' table instead of using a shortened 'obj_short' table.
        - BTree: data is moved from '_obj' to a new '_data' btree.
      
      - Undo is optimized so that backpointers are not required anymore to fetch data:
        - The checksum of an object is None only when creation is undone.
        - Removed DatabaseManager methods: _getObjectData, _getDataTIDFromData
        - DatabaseManager: move some code from _getDataTID to findUndoTID so that
          _getDataTID only has what's specific to backend.
      
      - Removed because already covered by ZODB tests:
        - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTID
        - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTIDFromData
      d5c469be
  30. 16 Sep, 2011 1 commit
  31. 10 Jun, 2011 1 commit
    • Julien Muchembled's avatar
      Introduce light functional tests, using threads and serialized processing · 1ef149c2
      Julien Muchembled authored
      This allows to setup an almost fully functional cluster without additional
      processes. Threads are scheduled so that they never run simultaneously,
      eliminating most random.
      There's still much improvement possible like controlled randomization,
      or easier debugging when switching from one thread to another.
      
      As mock objects are not usable in such tests, an API should be implemented to
      trace/count any method call we'd like to check.
      
      This fixes test_notifyNodeInformation_checkUnregisterStorage
      
      git-svn-id: https://svn.erp5.org/repos/neo/trunk@2775 71dcc9de-d417-0410-9af5-da40c76e7ee4
      1ef149c2
  32. 30 May, 2011 1 commit