- 27 Dec, 2016 1 commit
-
-
Julien Muchembled authored
-
- 26 Dec, 2016 5 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
-
Julien Muchembled authored
-
Julien Muchembled authored
-
Julien Muchembled authored
-
- 23 Dec, 2016 1 commit
-
-
Julien Muchembled authored
-
- 22 Dec, 2016 1 commit
-
-
Julien Muchembled authored
-
- 21 Dec, 2016 3 commits
-
-
Julien Muchembled authored
This fixes the following case when the backup is far behing the upstream DB, and there are transactions being committed at the same time: 1. replicate partition 0 2. replicate partition 0 3. replicate partition 1 4. replicate partition 0 5. replicate partition 1 6. replicate partition 2 7. replicate partition 0 ... and so on in a quadratic way. When the upstream activity was too high, the backup could even be stuck looping on the first partitions.
-
Julien Muchembled authored
The issue happens when there were commits while the backup cluster was down. In this case, the master thinks that these commits are already replicated, reporting wrong backup_tid to neoctl. It solved by itself once: - there are new commits triggering replication for all partitions; - all storage nodes have really replicated. This also resulted in an inconsistent database when leaving backup mode during this period.
-
Julien Muchembled authored
-
- 20 Dec, 2016 1 commit
-
-
Julien Muchembled authored
-
- 06 Dec, 2016 2 commits
-
-
Julien Muchembled authored
A backup master crashed with the following traceback after a reconnection: Traceback (most recent call last): File "neo/master/app.py", line 127, in run self._run() File "neo/master/app.py", line 147, in _run self.playPrimaryRole() File "neo/master/app.py", line 348, in playPrimaryRole self.backup_app.provideService()) File "neo/master/backup_app.py", line 123, in provideService poll(1) File "neo/lib/event.py", line 126, in poll to_process.process() File "neo/lib/connection.py", line 500, in process self._handlers.handle(self, self._queue.pop(0)) File "neo/lib/connection.py", line 110, in handle self._handle(connection, packet) File "neo/lib/connection.py", line 125, in _handle handler.packetReceived(connection, packet) File "neo/lib/handler.py", line 117, in packetReceived self.dispatch(*args) File "neo/lib/handler.py", line 66, in dispatch method(conn, *args, **kw) File "neo/master/handlers/backup.py", line 52, in invalidateObjects app.invalidatePartitions(tid, partition_set) File "neo/master/backup_app.py", line 257, in invalidatePartitions self.triggerBackup(node) File "neo/master/backup_app.py", line 281, in triggerBackup assert cell_list, offset AssertionError: 0
-
Julien Muchembled authored
-
- 01 Dec, 2016 5 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
Many "unit" tests (!= "threaded" tests) don't do more than checking implementation details, and increase coverage artificially. As with testEvent in commit 71e30fb9, most of these tests will either be removed or rewritten as threaded tests. The fact that the remaining unit tests actually cover code that other test don't gives motivation to maintain them. It will be also less code to update when switching to https://pypi.python.org/pypi/mock I proceeded as follows: 1. Measure coverage for all tests except unit tests. While checking my work, I found that coverage stats for threaded/functional/zodb tests are quite unstable, so I restarted from the beginning by doing this measure several times and only keeping the intersection of coverage data. 2. Measure coverage individually for each 'unit' tests, and substract the each result with the data in 1. 3. The candidates for deletion are those without any code covered. Tests I didn't delete: - neo.tests.master.testElectionHandler: I always do minimal changes about election, as long as there's no serious review. - neo.tests.master.testMasterPT.MasterPartitionTableTests.test_13_outdate - 4 tests in neo.tests.testPT: test_01_Cell, test_04_removeCell, test_06_clear, test_08_filled - neo.tests.storage.testStorage{MySQL,SQLite} - neo.tests.testUtil.UtilTests.testReadBufferRead In a way, this commit is actually quite conservative. There are still many useless tests that only check error paths and for simple tested methods, this is just duplicating thie tested code.
-
Julien Muchembled authored
-
Julien Muchembled authored
-
Julien Muchembled authored
-
- 30 Nov, 2016 3 commits
-
-
Julien Muchembled authored
- format IPv6 inside [] when followed by :<port> - unify rendering of node lists - neoctl: do not crash on empty DB (no PT id)
-
Julien Muchembled authored
-
Julien Muchembled authored
This fixes a reqression in commit c39d5c67, which could leads to failures like: 2016-11-29 09:56:58,756 ERROR ZODB.Connection Couldn't load state for 0x4843 Traceback (most recent call last): File "ZODB/Connection.py", line 860, in setstate self._setstate(obj) File "ZODB/Connection.py", line 901, in _setstate p, serial = self._storage.load(obj._p_oid, '') File "neo/client/Storage.py", line 82, in load return self.app.load(oid)[:2] File "neo/client/app.py", line 352, in load data, tid, next_tid, _ = self._loadFromStorage(oid, tid, before_tid) File "neo/client/app.py", line 372, in _loadFromStorage for node, conn in self.cp.iterateForObject(oid, readable=True): File "neo/client/pool.py", line 91, in iterateForObject pt = self.app.pt File "neo/client/app.py", line 146, in __getattr__ return self.__getattribute__(attr) AttributeError: 'Application' object has no attribute 'pt'
-
- 28 Nov, 2016 4 commits
-
-
Julien Muchembled authored
This is similar to commit 7aecdada and for completeness, we also protect unlock the same way.
-
Julien Muchembled authored
For 2 previous commits, we didn't reindent in order to keep the diff readable.
-
Julien Muchembled authored
-
Julien Muchembled authored
-
- 27 Nov, 2016 11 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
The added test describes how the new id timestamps fix the race condition. These timestamps could be any unique opaque values, and the protocol is extended to exchange them along with node ids. Internally, nodes also reuse timestamps as a marker to identify the first NotifyNodeInformation packets from the master: since this packet is a complete list of nodes in the cluster, any other node in the node manager has left the cluster definitely and is removed. The secondary masters didn't receive update about master nodes. It's also useless to send them information about non-master nodes.
-
Julien Muchembled authored
-
Julien Muchembled authored
When Client (including backup master) and admin nodes are identified, the primary master now sends them automatically all nodes with NotifyNodeInformation, as with storage nodes.
-
Julien Muchembled authored
- check address conflicts - on invalid values, reject peer instead of dying
-
Julien Muchembled authored
Listing connected/connecting nodes with a UUID is used: - in one place by storage nodes: here, it does not matter if we skip nodes that aren't really identified - in many places by the master, only for server connections, in which case we have equivalence with real identification So in practice, NodeManager is only simplified to reuse the 'identified' property of nodes.
-
Julien Muchembled authored
-
Julien Muchembled authored
Therefore, a client node in the node manager is always RUNNING.
-
Julien Muchembled authored
Although the change applies to any node with a temporary ids (all but storage), only clients don't have addresses and are therefore not recognizable. After a client is disconnected from the master and before reconnecting, another client may join the cluster and "steals" the id of the first client. This issue leads to stuck clients, failing in loop with exceptions like the following one: ERROR ZODB.Connection Couldn't load state for 0x0251 Traceback (most recent call last): File "ZODB/Connection.py", line 860, in setstate self._setstate(obj) File "ZODB/Connection.py", line 901, in _setstate p, serial = self._storage.load(obj._p_oid, '') File "neo/client/Storage.py", line 82, in load return self.app.load(oid)[:2] File "neo/client/app.py", line 353, in load data, tid, next_tid, _ = self._loadFromStorage(oid, tid, before_tid) File "neo/client/app.py", line 373, in _loadFromStorage for node, conn in self.cp.iterateForObject(oid, readable=True): File "neo/client/pool.py", line 91, in iterateForObject pt = self.app.pt File "neo/client/app.py", line 145, in __getattr__ self._getMasterConnection() File "neo/client/app.py", line 214, in _getMasterConnection result = self.master_conn = self._connectToPrimaryNode() File "neo/client/app.py", line 246, in _connectToPrimaryNode handler=handler) File "neo/lib/threaded_app.py", line 154, in _ask _handlePacket(qconn, qpacket, kw, handler) File "neo/lib/threaded_app.py", line 135, in _handlePacket handler.dispatch(conn, packet, kw) File "neo/lib/handler.py", line 66, in dispatch method(conn, *args, **kw) File "neo/lib/handler.py", line 188, in error getattr(self, Errors[code])(conn, message) File "neo/client/handlers/__init__.py", line 23, in protocolError raise StorageError("protocol error: %s" % message) StorageError: protocol error: already connected
-
Julien Muchembled authored
-
Julien Muchembled authored
-
- 25 Nov, 2016 2 commits
-
-
Julien Muchembled authored
-
Julien Muchembled authored
-
- 21 Nov, 2016 1 commit
-
-
Julien Muchembled authored
`ClientCache._oid_dict` shall not have empty values. For a given oid, when the last item is removed from the cache, the oid must be removed as well to free memory. In some cases, this was not done. A consequence of this bug is the following exception: ERROR ZODB.Connection Couldn't load state for 0x02d1e1e4 Traceback (most recent call last): File "ZODB/Connection.py", line 860, in setstate self._setstate(obj) File "ZODB/Connection.py", line 901, in _setstate p, serial = self._storage.load(obj._p_oid, '') File "neo/client/Storage.py", line 82, in load return self.app.load(oid)[:2] File "neo/client/app.py", line 358, in load self._cache.store(oid, data, tid, next_tid) File "neo/client/cache.py", line 228, in store prev = item_list[-1] IndexError: list index out of range
-