Skip to content

N
neoppod
An error occurred fetching the project authors.
Switch branch/tag
  1. 17 Jan, 2012 1 commit
  3. 11 Oct, 2011 2 commits
    • Julien Muchembled's avatar
      Fix protocol and DB schema so that storages can handle transactions of any size · d5c469be
      Julien Muchembled authored 
      - Change protocol to use SHA1 for all checksums:
  - Use SHA1 instead of CRC32 for data checksums.
  - Use SHA1 instead of MD5 for replication.

- Change DatabaseManager API so that backends can store raw data separately from
  object metadata:
  - When processing AskStoreObject, call the backend to store the data
    immediately, instead of keeping it in RAM or in the temporary object table.
    Data is then referenced only by its checksum.
    Without such change, the storage could fail to store the transaction due to
    lack of RAM, or it could make tpc_finish step very slow.
  - Backends have to store data in a separate space, and remove entries as soon
    as they get unreferenced. So they must have an index of checksums in object
    metadata space. A new '_uncommitted_data' backend attribute keeps references
    of uncommitted data.
  - New methods: _pruneData, _storeData, storeData, unlockData
  - MySQL: change vertical partitioning of 'obj' by having data in a separate
    'data' table instead of using a shortened 'obj_short' table.
  - BTree: data is moved from '_obj' to a new '_data' btree.

- Undo is optimized so that backpointers are not required anymore to fetch data:
  - The checksum of an object is None only when creation is undone.
  - Removed DatabaseManager methods: _getObjectData, _getDataTIDFromData
  - DatabaseManager: move some code from _getDataTID to findUndoTID so that
    _getDataTID only has what's specific to backend.

- Removed because already covered by ZODB tests:
  - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTID
  - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTIDFromData
      d5c469be
    • Julien Muchembled's avatar
      Allow NEO to store empty values · d90c5b83
      Julien Muchembled authored 
      This changes how NEO stores undo information
and how it is transmitted on the network.
      d90c5b83
  5. 08 Sep, 2011 1 commit
  7. 07 Sep, 2011 1 commit
  9. 24 Jun, 2011 1 commit
    • Julien Muchembled's avatar
      client: do not clear the entire connection cache in tpc_finish ! · b55075f7
      Julien Muchembled authored 
      This fixes a regression introduced by implementation of IMVCCStorage in r2532.
On recent ZODB, this fixes a severe performance issue.
With ZODB 3.4, objects were never invalidated, which was even worse.

This fixes includes compatibility code for ZODB 3.4 so that each connection
has its own NEOStorage instance.
The DB's storage is changed to always consider the last revision of
objects.

IMVCCStorage seems too complicated to me. Connection should be a better
place to implement it (by extending/fixing the 'before' attribute).
So we have decided that NEOStorage stops implementing IMVCCStorage.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2802 71dcc9de-d417-0410-9af5-da40c76e7ee4
      b55075f7
  11. 14 Jun, 2011 2 commits
  13. 12 Apr, 2011 1 commit
    • Julien Muchembled's avatar
      Rewrite storage cache of client · edde1fe3
      Julien Muchembled authored 
      - Stop using a list of (in)validated tid (hence removal of RevisionIndex),
  because it can't work in all cases and would even cause memory leaks.
  For example, this bug could lead to ConflictError with a single client.
  Fixit it also requires that database backends always return the next serial.
- Several performance improvements. The most important one is when the latest
  version of an object is cached: it inherits the access counter of the
  previous one (for the same oid), which gets in turn its counter reset.
- Do not waste CPU evaluating the real size taken by an entry in memory.
  Just use 'len' on the value (which is always a pickle data, i.e. a string).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2711 71dcc9de-d417-0410-9af5-da40c76e7ee4
      edde1fe3
  15. 17 Jan, 2011 1 commit
  17. 06 Jan, 2011 1 commit
    • Vincent Pelletier's avatar
      After undoing an object, its current serial goes back in time. · 10c96d61
      Vincent Pelletier authored 
      This is important for chaining "undo" for a given object in a single
transaction, to allow checking for conflicts.
Also, this means that get_baseTID return value cannot be used to update
cache when undoing, so only update cache when storing a new revision.
Actually, cache was never updated when undoing anyway, as current snapshot
TID was unlikely to exist as a serial for any object, so it would not be
found in cache and update would be a no-op.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2597 71dcc9de-d417-0410-9af5-da40c76e7ee4
      10c96d61
  19. 05 Jan, 2011 1 commit
  21. 14 Dec, 2010 1 commit
    • Vincent Pelletier's avatar
      Implement MVCC. · 4e402dda
      Vincent Pelletier authored 
      Remove round-trip to master upon "load" call.
Move load/loadBefore/loadSerial/loadEx from app.py to Storage.py.
This is required to get rid of master node round-trip upon each "load"
call.
Get rid of no-op-ish "sync" implementation.
Separate "undoing transaction ID" from "undoing transaction database
snapshot" when undoing.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2532 71dcc9de-d417-0410-9af5-da40c76e7ee4
      4e402dda
  23. 07 Dec, 2010 1 commit
  25. 17 Nov, 2010 2 commits
  27. 01 Oct, 2010 2 commits
  29. 24 Sep, 2010 1 commit
  31. 23 Sep, 2010 1 commit
  33. 05 Sep, 2010 6 commits
  35. 03 Sep, 2010 1 commit
  37. 02 Sep, 2010 1 commit
    • Vincent Pelletier's avatar
      Make undo implementation work with replication. · 7835c09e
      Vincent Pelletier authored 
      The problem with previous implementation was that each storage locally
decided what undo actually did to data. This causes problems when a
storage doesn't have a complete view of past transaction but accepts write
queries, ie when it replicates.
This implementation reduces the decision to a readable subset of storage
nodes (which are hence not replicating), and then sends that decision to
all storage nodes, hence fixing the issue.
Also, DatabaseManager.storeTransaction now consistently expects object's
value_serial to be packed, not an integer.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2285 71dcc9de-d417-0410-9af5-da40c76e7ee4
      7835c09e
  39. 24 Aug, 2010 1 commit
    • Vincent Pelletier's avatar
      Improve replication SQL queries. · d8a7a177
      Vincent Pelletier authored 
      It is more efficient to provide a boundary value than a row count range.
This fixes replication on partitions with a large number of objects, revisions
or transactions: query time is now constant where it used to increase, causing
timeout problems when query duration exceeded ping time + ping timeout (11s
currently).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2221 71dcc9de-d417-0410-9af5-da40c76e7ee4
      d8a7a177
  41. 18 Jun, 2010 1 commit
  43. 13 May, 2010 2 commits
  45. 26 Mar, 2010 1 commit
    • Vincent Pelletier's avatar
      Use value_serial for undo support. · 83c02447
      Vincent Pelletier authored 
      This mimics what FileStorage uses (file offsets) but in a relational manner.
This offloads decision of the ability to undo a transaction to storages,
avoiding 3 data loads for each object in the transaction at client side.
This also makes Neo refuse to undo transactions where object data would happen
to be equal between current value and undone value.
Finally, this is required to make database pack work properly (namely, it
prevents loosing objects which are orphans at pack TID, but are reachable
after it thanks to a transactional undo).

Also, extend storage's transaction manager so database adapter can fetch data
already sent by client in the same transaction, so it can undo multiple
transactions at once. Requires making object lock re-entrant (done in this
commit).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@1978 71dcc9de-d417-0410-9af5-da40c76e7ee4
      83c02447
  47. 25 Feb, 2010 2 commits
  49. 23 Feb, 2010 1 commit
  51. 08 Feb, 2010 2 commits
  53. 01 Feb, 2010 1 commit
  55. 20 Jan, 2010 1 commit
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7