nexedi / neoppod · Commits

This reverts commit d3c22487 partially
and fixes the bug in a much simpler way.

Traceback (most recent call last):
  [...]
  File "neo/storage/handlers/client.py", line 115, in askStoreObject
    *e.args)
  File "neo/lib/handler.py", line 333, in queueEvent
    self.sortQueuedEvents()
  File "neo/lib/handler.py", line 326, in <lambda>
    self._event_queue.sort(key=key))()
  File "neo/storage/transactions.py", line 67, in __lt__
    return self.locking_tid < other.locking_tid
AttributeError: 'NoneType' object has no attribute 'locking_tid'

Pending events:
  (None, <askFetchTransactions: ...>)
  (<Transaction(C13, locking_tid=03c266508a058388, tid=None, age=0.21s) at 0x7f086bbc3d50>, <_askStoreObject: ...>)

It was out-of-date since commit 23b6a66a.

This time we had issues with _getObject and getReplicationObjectList
on production.
It seems related to the upgrade of MariaDB, from 10.0.21 to 10.1.25.

See also commits eaa00a88
             and 13911ca3

The explanation became wrong during a git-rebase, when it was decided
to keep the old code that drop partitions. The new one needs more work
and it is kept in a branch.

Using NEO 0.9.1 and partitioning enabled, I could reproduce the issue with
MariaDB 5.3.5, but not with MariaDB 5.3.12 and 5.5.23. I suppose it was fixed.

In testOudatedCellsOnDownStorage,
  'select count(*) from obj' returned a wrong value (always 1).

Strangely, 'select count(*) from test_neo0.obj' was always correct (102).

When 'neo.tests.cluster' is loaded (usually when functional tests are run),
__builtin__.pdb is replaced by an extended pdb, which should behave the same
way if it is used like the former.

winpdb is so slow that a console pdb is often preferred.

The most important change is that it does not discard readable cells too
quickly anymore. A partition can now have multiple FEEDING cells, to avoid
going below the wanted level of replication.

The new algorithm is also better at minimizing the amount replication.

The MySQL implementation is written to work around the issue reported at
  https://jira.mariadb.org/browse/MDEV-12867

Since it's not worth anymore to keep track of the last connection activity
(which, btw, ignored TCP ACKs, i.e. timeouts could theorically be triggered
before all the data were actually sent), the semantics of closeClient has also
changed. Before this commit, the 1-minute timeout was reset whenever there was
activity (connection still used as server). Now, it happens exactly 100 seconds
after the connection is not used anymore as client.

The next line (MTClientConnection) already logs new connections
and the storage node is necessarily in RUNNING state.

Now, the primary master is the running master with None displayed in the
last column. Before, it could be the id timestamp of when it was secondary,
which was obsolete information.

This fixes up commit 23b6a66a,
which reimplements election.

  poll raised, retrying
  Traceback (most recent call last):
    ...
    File "neo/client/handlers/master.py", line 41, in notPrimaryMaster
      super(PrimaryNotificationsHandler, self).notPrimaryMaster(*args)
    File "neo/lib/handler.py", line 157, in notPrimaryMaster
      assert primary != self.app.server
    File "neo/client/app.py", line 109, in __getattr__
      return self.__getattribute__(attr)
  AttributeError: 'Application' object has no attribute 'server'

This fixes the following crash:

  Traceback (most recent call last):
    ...
    File "neo/master/handlers/identification.py", line 94, in requestIdentification
      uuid = app.getNewUUID(uuid, address, node_type)
    File "neo/master/app.py", line 449, in getNewUUID
      assert uuid != self.uuid
  AssertionError

This really fixes the bug described in
commit 40bac312,
which could probably be reverted. It only reduced the probability of failure.

What happened is that the second conflict on 'a' for t3 what first reported by
an answer to first store with:
- a base serial at which a=0
- a conflict serial at which a=7
However, the cached data is not 8 anymore but 12, since a second store already
occurred after the first conflict (reported by the other storage node).

When this conflict was resolved before receiving the conflict for second store,
it gave:

  resolve(old=0, saved=7, new=12) -> 19

instead of:

  resolve(old=4, saved=7, new=12) -> 15

(if we still had the data of the first store, we could also do
  resolve(old=0, saved=7, new=8)
 but that would be inefficient from a memory point of view)

The bug was difficult to reproduce. testNotifyReplicated had to be run many
many times before that race conditions trigger it. The test was changed to
enforce some of them, and the above scenario now happens almost always.