neoppod:745b7616a860606af63ea1f1578562a3babd0f29 commitshttps://lab.nexedi.com/nexedi/neoppod/-/commits/745b7616a860606af63ea1f1578562a3babd0f292014-07-30T14:42:35+02:00https://lab.nexedi.com/nexedi/neoppod/-/commit/745b7616a860606af63ea1f1578562a3babd0f29NEO 1.22014-07-30T14:42:35+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/233a50cc70473ef6dab803a52833c4d1b6b04935storage: do not commit too often when importing many small transactions2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.com
The only purpose of committing during import is to not restart from the
beginning in case of interruption.https://lab.nexedi.com/nexedi/neoppod/-/commit/6ca0ab7051961b922a360a78eff025507d148850The implementation of some methods in storage backend is optional2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.com
If not implemented, this should not result in a fatal error that kills the
storage. Instead of that, a new error packet is sent so that the exception is
raised in the client.https://lab.nexedi.com/nexedi/neoppod/-/commit/de4c9e1da686198457f9d442cc0e9244990e4226New 'Importer' storage backend2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/53df0e1a5953cb72ea3e18c149307428404af2cetypos2014-07-25T19:08:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/55a60277442bc3200161acd0000323477c20416fAdd warning about possible misuse of replicas2014-07-25T19:08:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/d82383ef98ccb3c55ad28714be3a7861d4f136d7Raise verbosity of 'simple' script2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
That seems too verbose now but it was difficult to know the ports of nodes
when only warnings were output.https://lab.nexedi.com/nexedi/neoppod/-/commit/7e89973c569d48bfbcdc3be19a933454d58c9e7cmaster: do not die if neoctl asks the master directly for cluster state2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
Of course, neoctl should only connect to the admin, but in case of user mistake
the queried node should not die. Ideally, all other commands should be checked.https://lab.nexedi.com/nexedi/neoppod/-/commit/d90208a323df32ab8d8152f218bd54e5da4b2bc3Fix neomigrate command2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
It was broken since review of logging
(commit <a href="/levin.zimmermann/neoppod/-/commit/1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-original="1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-link="false" data-link-reference="false" data-project="1572" data-commit="1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Review logging to keep all debugging information in RAM and flush only if useful" class="gfm gfm-commit has-tooltip">1fce5cc4</a>).https://lab.nexedi.com/nexedi/neoppod/-/commit/bced2e7b7a2087f818770d2c293d2b27e6ca8ed1storage: revert "remake _getPartition lazy, similarly to fallback decorator"2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
_getPartition optimization can't be done at class level and properties hide
instance attribute so implement laziness in __getattr__https://lab.nexedi.com/nexedi/neoppod/-/commit/ea5cdd0bf21ffaceaf5acd8eb1cb145faf5eff82storage: revert addition of 'app' parameter in setup()2014-07-22T12:09:13+02:00Julien Muchembledjm@nexedi.com
This is not a valid place to call app.newTaskhttps://lab.nexedi.com/nexedi/neoppod/-/commit/c2cc0c11586f1e331c349619ec4edac076135616mysql: fail instead of silently reconnect if there is any pending change2014-07-08T20:55:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/069b95e5c2d7322f5ba1139230443bb31f1e6936mysql: small optimization by making 'escape' a property2014-07-08T20:54:27+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/3e91d2696adafbe69d51bef6f5973fc167981c01storage: use defaultdict to track data of uncommitted objects2014-07-08T20:54:27+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/a0bd2ae86ce7adb72bd4ed92ebbed9ba5c8eae8bstorage: fix memory leak in replication2014-07-08T20:54:19+02:00Julien Muchembledjm@nexedi.com
The following 3 methods are renamed:
unlockData -> releaseData
storeData -> holdData
_storeData -> storeData
and StorageOperationHandler use the new storeData instead of the old one.https://lab.nexedi.com/nexedi/neoppod/-/commit/f1b72dfe94f8b19517ac9ebdfaa75687f2fc161bstorage: remake _getPartition lazy, similarly to fallback decorator2014-07-04T20:14:07+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/e823b93c9b7b31746fafde2f65ae80cdf8bbc753tests: compression can be in threaded clusters2014-07-04T19:25:35+02:00Julien Muchembledjm@nexedi.com
For the moment, this is only useful for debugging.https://lab.nexedi.com/nexedi/neoppod/-/commit/0accf1107442a765b9ef010f686b428b13958575storage: backend optimizations2014-07-04T19:25:27+02:00Julien Muchembledjm@nexedi.com
Transaction manager uses a new getLastObjectTID method instead of
getObjectHistory, because the latter fetches the data and this is
a waste of time here.
Similarly, minimize SQL work in getObject if the object can't be found.https://lab.nexedi.com/nexedi/neoppod/-/commit/edb1b8669df246336de4e79a7f0ae3534e245cd8storage: faster & reusable "log-once" for fallback backend methods2014-07-04T19:18:31+02:00Julien Muchembledjm@nexedi.com
Although it's a warning for developers, it's only informational for users
so lower severity in order not to frighten them.https://lab.nexedi.com/nexedi/neoppod/-/commit/0b8bfba3815eba96df4a44a828b8849fa4e16e05storage: do not recreate tables when clearing database at exit2014-07-04T18:53:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/1745d6ac54c5dbaf9a2fd9eae52399cc864a5b33storage: optimize DatabaseManager._getPartition2014-06-26T18:17:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/07e11994379eb2975ec75d437c63262932cb2748storage: small optimisation of askObject for most likely code path2014-06-24T19:34:22+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/84f9894e68b8a5683db32bb7bc2ee417f4080df3storage: review search of data serial for undo2014-06-24T19:33:19+02:00Julien Muchembledjm@nexedi.com
- Fix case of empty values: there's no reason to do anything special for them.
- Do not warn about multiple level of indirection to get data serial.
With the current structure of tables, this does not cause significant
performance issue as it did before.https://lab.nexedi.com/nexedi/neoppod/-/commit/d562bf8ffef5f0464575a33223ead029dce2bf21client: clean up import/export code2014-06-20T19:31:21+02:00Julien Muchembledjm@nexedi.com
Export:
- Remove leftover warning about a bug that was fixed in
commit <a href="/kirr/neo/-/commit/e76af2977448062569cb5f95b6b625d9583efb29" data-original="e76af2977448062569cb5f95b6b625d9583efb29" data-link="false" data-link-reference="false" data-project="73" data-commit="e76af2977448062569cb5f95b6b625d9583efb29" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="client: in iterator records, export data serial as stored by NEO" class="gfm gfm-commit has-tooltip">e76af297</a>
- In neomigrate script, open NEO storage read-only.
- IStorageIteration is already implemented.
Import:
- Review comments.
- In neomigrate script, warn that IStorageRestoreable is not implemented.
- Do not call 'close' method on source iterator. BaseStorage does not do it and
this is not part of ZODB API. In the case of FileStorage, resource are freed
automatically during garbage collection.https://lab.nexedi.com/nexedi/neoppod/-/commit/fcff26db9447b3c49924ce941a6ae10ec7d3bc35Use several partitions in functional client tests2014-06-20T12:09:51+02:00Julien Muchembledjm@nexedi.com
This is more realistic than testing with a single partition, in particular when
there are more storage nodes that copies.https://lab.nexedi.com/nexedi/neoppod/-/commit/1d11287f533915396b2838a2cf53a60c86a82cf5storage: more refactoring of backends2014-06-20T11:24:15+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/e76af2977448062569cb5f95b6b625d9583efb29client: in iterator records, export data serial as stored by NEO2014-06-20T01:51:10+02:00Julien Muchembledjm@nexedi.com
There is simply no way to guess data serials and instead of producing random
values, the only solution is to retrieve the values from storages.
There are still differences for data serials between FileStorage and NEO:
- NEO always resolves to original serial, to avoid any indirection
(which slightly speeds up undo at the expense of a more complex pack code)
- NEO does not make any difference between object deletion and creation undone
(data serial always null in storage)
It has to be decided whether NEO implementation should be changed about this.
Apart from that, conversion database back from NEO should be fixed.
testExportFileStorageBug passes and there was in fact no FileStorage bug.
Another change is that iterator does not trash the client cache anymore.https://lab.nexedi.com/nexedi/neoppod/-/commit/441145e5c288415f0734e52d53b264654ac3fe65client: simplify iterator2014-06-19T18:04:24+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/bd03b14b5e03ba529cb6ee117a5a16c7fc858ec1storage: refactoring & cleanup2014-06-19T16:44:55+02:00Julien Muchembledjm@nexedi.com
- _[gs]etPackTID accessors implementation is not backend-specific
so move them to superclass
- _getObjectLength method is useless since data_tid always contains the wanted
information, regardless the contents of value_tid columnhttps://lab.nexedi.com/nexedi/neoppod/-/commit/d63e45d343c9520c06596ecdac569e11476297e4storage: lazy import of backends2014-06-19T16:28:16+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/ca0afaa3cfc14b51a45ff23dc63f79a1e71eb626tests: fix random failure in testStoppingPrimaryWithOneSecondary2014-06-05T11:35:45+02:00Julien Muchembledjm@nexedi.com
Sometimes, the tested cluster reacts so quickly that a new primary master arised
before we test that at some point, there is no primary master.https://lab.nexedi.com/nexedi/neoppod/-/commit/9fe344397d92bc9150d504382ea0f99935427a93Fix tests broken by API changes in new psutil 2.0.02014-06-04T22:52:35+02:00Julien Muchembledjm@nexedi.com
This fixes:
Traceback (most recent call last):
File "neo/tests/functional/testMaster.py", line 50, in testStoppingSecondaryMaster
self.neo.expectDead(master)
File "neo/tests/functional/__init__.py", line 615, in expectDead
self.expectCondition(callback, *args, **kw)
File "neo/tests/functional/__init__.py", line 509, in expectCondition
'History: %s' % opaque_history)
AssertionError: Timeout while expecting condition. History: [False, False, False, False, False, False, False, False, False, False, False]https://lab.nexedi.com/nexedi/neoppod/-/commit/ffe346004d03286a752f9a6ef7ce03927f70d5a0fixup! client: drop support for ZODB < 3.102014-06-04T10:24:19+02:00Julien Muchembledjm@nexedi.com
See commit <a href="/kirr/neo/-/commit/d9ab77b8347145c41d99a52aa5677ea60312986a" data-original="d9ab77b8347145c41d99a52aa5677ea60312986a" data-link="false" data-link-reference="false" data-project="73" data-commit="d9ab77b8347145c41d99a52aa5677ea60312986a" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="client: drop support for ZODB < 3.10" class="gfm gfm-commit has-tooltip">d9ab77b8</a>https://lab.nexedi.com/nexedi/neoppod/-/commit/9a6ad2e61a41ecc7abc1e975fa7dbe75af2d25f9Update BUGS2014-06-03T19:33:36+02:00Julien Muchembledjm@nexedi.com
One entry should have been removed before v1.1https://lab.nexedi.com/nexedi/neoppod/-/commit/a5f2f60459878aae903129cda1d2f444b625cdd0tests: review report and mark known failures as expected2014-06-03T19:33:36+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/d250deca6009bbb0daa88d399db0ca407d618032Drop support for Python < 2.72014-06-03T19:33:36+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/14fd9cd3676d533008d40bfad387765d9bca19aaTODO: ctl command to truncate DB + hint for better performance2014-05-29T20:01:43+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/5ce3ad7018de9f9addae3a4da51def04172e2877storage: add support for recent version of sqlite2014-01-08T19:07:40+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/d4ed0828eeebab95563750b3fed81b86f52e224fNEO 1.12014-01-07T16:44:19+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/fd4cfaa9233cdabfd8f0bac24a4ebc9bc5f055f6Add test showing that clients may be stuck on an old snapshot in case of fail...2014-01-07T15:32:35+01:00Julien Muchembledjm@nexedi.com
If anything wrong happens after a transaction is locked and before the end of
onTransactionCommitted, recovery phase should be run again, so that the master
gets correct last tid.
Following patch by Vincent is an attempt to fix this:
--- a/neo/master/app.py
+++ b/neo/master/app.py
@@ -329,8 +329,8 @@ def playPrimaryRole(self):
# recover the cluster status at startup
try:
- self.runManager(RecoveryManager)
while True:
+ self.runManager(RecoveryManager)
self.runManager(VerificationManager)
try:
if self.backup_tid:
@@ -338,10 +338,6 @@ def playPrimaryRole(self):
raise RuntimeError("No upstream cluster to backup"
" defined in configuration")
self.backup_app.provideService()
- # Reset connection with storages (and go through a
- # recovery phase) when leaving backup mode in order
- # to get correct last oid/tid.
- self.runManager(RecoveryManager)
continue
self.provideService()
except OperationFailure: