neoppod:53887d79c385aa8871e9c97ded545a24d39a7c0f commitshttps://lab.nexedi.com/nexedi/neoppod/-/commits/53887d79c385aa8871e9c97ded545a24d39a7c0f2015-05-05T15:02:49+02:00https://lab.nexedi.com/nexedi/neoppod/-/commit/53887d79c385aa8871e9c97ded545a24d39a7c0fmysql: new configuration parameter to choose engine (InnoDB or TokuDB)2015-05-05T15:02:49+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/2a3392eb1595f4186c15f31a9a8d8de92110aabaimporter: merge all inserts to 'obj' into a single request per transaction2015-04-26T18:49:24+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/cd33de9fe5bdb1ec624d1d32996b6431936348f0importer: speed up txn lookup when migrating from FileStorage2015-04-26T18:21:28+02:00Julien Muchembledjm@nexedi.com
This speeds up 2 operations that were horribly slow (linear scan
of source database, from the beginning or from the end):
- read access to not yet migrated data
- resumption (restart of storage node whereas import was not finshed)
Testing will tell if this patch is enough, or if more efficient solutions are
required, like:
- building a preliminary {tid->file_pos} index while the source DB is still
in use (in this case, NEO may have to deal with a small gap at the end)
- disabling ZODB features requiring data_serial (undo)https://lab.nexedi.com/nexedi/neoppod/-/commit/e582696c1ef3224fa55d37dbd57a0bf6b0f37b21NEO 1.32015-01-13T19:03:45+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/49b6a83add54a54c290e00a86bdaf004ec7cc51dmaster: add command line options for upstream cluster/masters2014-12-09T16:15:50+01:00Julien Muchembledjm@nexedi.com
Before it was only possible to specify them in a configuration file.https://lab.nexedi.com/nexedi/neoppod/-/commit/acf91be86533665f1c5a93e8958b2baa0eaa04d6Add note about broken 'wait' option in configuration file2014-12-05T19:59:48+01:00Julien Muchembledjm@nexedi.com
Value must be cast as integer.
The whole configuration part should be reviewed to use argparse
and avoid duplicated code.https://lab.nexedi.com/nexedi/neoppod/-/commit/bc70a7537e99e8f272785296b599df86c919f5d4neolog: do not spam the console when piped to a process that exits prematurely2014-12-05T19:50:21+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/9bd318038fd4d9697f469abd7f09b12a1fa6e85fneolog: add support for gzip/bz2 compressed logs2014-12-05T19:40:23+01:00Julien Muchembledjm@nexedi.com
They're decompressed automatically to temporary files.https://lab.nexedi.com/nexedi/neoppod/-/commit/1b1a9bcd188c2d67d7dc9624bba8060a24d81a96importer: when mapping oids, repickle without loading any object2014-12-05T17:07:07+01:00Julien Muchembledjm@nexedi.com
The use of ZODB.broken was completely wrong so previous code only worked
if NEO could import all classes used by the application.https://lab.nexedi.com/nexedi/neoppod/-/commit/764b41c07f5f98bdb05e476ed6ed82baa3bf7de3importer: fix resuming2014-12-05T12:20:55+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/58cf9e2410d0e1bdf27bffce40e8a7e1b1289498client: document default value of 'compress' option2014-12-05T12:09:21+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/9af7ecab22d7e33ad1b0d048ff982786ab51dafamysql: quote `partition` column for compatibility with recent MariaDB (>=10)2014-11-07T19:42:48+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/059cb03cb1c665077672c7bbe26fb8b79f335b15Respect NEO_TESTS_ADAPTER for all tests run by -u2014-11-07T19:42:48+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/3fbffa00c37f78fb10b1a112e493d27404b97ffdFix test_parseNodeAddress when ipv6 is configured and preferred for 'localhost'2014-11-07T19:42:48+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/c21f054e500804469e477568a30c8c83d242158eclient: register log rotation when running in Zope2014-11-07T19:42:48+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/3428da3c48a434a1267ad457ceff715a568f4f76logger: new RTMIN+1 signal to reopen the log2014-11-07T19:42:48+01:00Julien Muchembledjm@nexedi.com
RTMIN+1 & RTMIN+2 signals, which were previously used for debugging,
have been remapped to RTMIN+2 & RTMIN+3
No journal on disk anymore because this is incompatible with the
rename+reopen way to rotate logs, and we want to support logrotate.
Of course in case of crash the log may be corrupt or entries may be lost,
but we currenty don't think NEO logs would be useful to debug such failure.https://lab.nexedi.com/nexedi/neoppod/-/commit/17fc0ef901bce04600caaaa5f6d6d61abc696088logger: new _setup() method to setup the logger when lock is already acquired2014-10-27T20:33:31+01:00Julien Muchembledjm@nexedi.com
This commit only moves code from setup() to _setup()https://lab.nexedi.com/nexedi/neoppod/-/commit/88c0d6f5aebc8db2caf9b2af9640f5698a8488f7Validate reStructuredText if docutils is available2014-07-30T15:06:56+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/745b7616a860606af63ea1f1578562a3babd0f29NEO 1.22014-07-30T14:42:35+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/233a50cc70473ef6dab803a52833c4d1b6b04935storage: do not commit too often when importing many small transactions2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.com
The only purpose of committing during import is to not restart from the
beginning in case of interruption.https://lab.nexedi.com/nexedi/neoppod/-/commit/6ca0ab7051961b922a360a78eff025507d148850The implementation of some methods in storage backend is optional2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.com
If not implemented, this should not result in a fatal error that kills the
storage. Instead of that, a new error packet is sent so that the exception is
raised in the client.https://lab.nexedi.com/nexedi/neoppod/-/commit/de4c9e1da686198457f9d442cc0e9244990e4226New 'Importer' storage backend2014-07-30T14:36:59+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/53df0e1a5953cb72ea3e18c149307428404af2cetypos2014-07-25T19:08:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/55a60277442bc3200161acd0000323477c20416fAdd warning about possible misuse of replicas2014-07-25T19:08:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/d82383ef98ccb3c55ad28714be3a7861d4f136d7Raise verbosity of 'simple' script2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
That seems too verbose now but it was difficult to know the ports of nodes
when only warnings were output.https://lab.nexedi.com/nexedi/neoppod/-/commit/7e89973c569d48bfbcdc3be19a933454d58c9e7cmaster: do not die if neoctl asks the master directly for cluster state2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
Of course, neoctl should only connect to the admin, but in case of user mistake
the queried node should not die. Ideally, all other commands should be checked.https://lab.nexedi.com/nexedi/neoppod/-/commit/d90208a323df32ab8d8152f218bd54e5da4b2bc3Fix neomigrate command2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
It was broken since review of logging
(commit <a href="/nexedi/neoppod/-/commit/1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-original="1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-link="false" data-link-reference="false" data-project="72" data-commit="1fce5cc419f9220962eda2f5dfd325293c5c6b8a" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Review logging to keep all debugging information in RAM and flush only if useful" class="gfm gfm-commit has-tooltip">1fce5cc4</a>).https://lab.nexedi.com/nexedi/neoppod/-/commit/bced2e7b7a2087f818770d2c293d2b27e6ca8ed1storage: revert "remake _getPartition lazy, similarly to fallback decorator"2014-07-24T17:46:09+02:00Julien Muchembledjm@nexedi.com
_getPartition optimization can't be done at class level and properties hide
instance attribute so implement laziness in __getattr__https://lab.nexedi.com/nexedi/neoppod/-/commit/ea5cdd0bf21ffaceaf5acd8eb1cb145faf5eff82storage: revert addition of 'app' parameter in setup()2014-07-22T12:09:13+02:00Julien Muchembledjm@nexedi.com
This is not a valid place to call app.newTaskhttps://lab.nexedi.com/nexedi/neoppod/-/commit/c2cc0c11586f1e331c349619ec4edac076135616mysql: fail instead of silently reconnect if there is any pending change2014-07-08T20:55:58+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/069b95e5c2d7322f5ba1139230443bb31f1e6936mysql: small optimization by making 'escape' a property2014-07-08T20:54:27+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/3e91d2696adafbe69d51bef6f5973fc167981c01storage: use defaultdict to track data of uncommitted objects2014-07-08T20:54:27+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/a0bd2ae86ce7adb72bd4ed92ebbed9ba5c8eae8bstorage: fix memory leak in replication2014-07-08T20:54:19+02:00Julien Muchembledjm@nexedi.com
The following 3 methods are renamed:
unlockData -> releaseData
storeData -> holdData
_storeData -> storeData
and StorageOperationHandler use the new storeData instead of the old one.https://lab.nexedi.com/nexedi/neoppod/-/commit/f1b72dfe94f8b19517ac9ebdfaa75687f2fc161bstorage: remake _getPartition lazy, similarly to fallback decorator2014-07-04T20:14:07+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/e823b93c9b7b31746fafde2f65ae80cdf8bbc753tests: compression can be in threaded clusters2014-07-04T19:25:35+02:00Julien Muchembledjm@nexedi.com
For the moment, this is only useful for debugging.https://lab.nexedi.com/nexedi/neoppod/-/commit/0accf1107442a765b9ef010f686b428b13958575storage: backend optimizations2014-07-04T19:25:27+02:00Julien Muchembledjm@nexedi.com
Transaction manager uses a new getLastObjectTID method instead of
getObjectHistory, because the latter fetches the data and this is
a waste of time here.
Similarly, minimize SQL work in getObject if the object can't be found.https://lab.nexedi.com/nexedi/neoppod/-/commit/edb1b8669df246336de4e79a7f0ae3534e245cd8storage: faster & reusable "log-once" for fallback backend methods2014-07-04T19:18:31+02:00Julien Muchembledjm@nexedi.com
Although it's a warning for developers, it's only informational for users
so lower severity in order not to frighten them.https://lab.nexedi.com/nexedi/neoppod/-/commit/0b8bfba3815eba96df4a44a828b8849fa4e16e05storage: do not recreate tables when clearing database at exit2014-07-04T18:53:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/1745d6ac54c5dbaf9a2fd9eae52399cc864a5b33storage: optimize DatabaseManager._getPartition2014-06-26T18:17:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/neoppod/-/commit/07e11994379eb2975ec75d437c63262932cb2748storage: small optimisation of askObject for most likely code path2014-06-24T19:34:22+02:00Julien Muchembledjm@nexedi.com