1. 19 Aug, 2020 3 commits
    • Julien Muchembled's avatar
      qa: skip broken ZODB test · a1418c9d
      Julien Muchembled authored
      ======================================================================
      FAIL: check_tid_ordering_w_commit (neo.tests.zodb.testBasic.BasicTests)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "ZODB/tests/BasicStorage.py", line 397, in check_tid_ordering_w_commit
          self.assertEqual(results.pop('lastTransaction'), tids[1])
        File "neo/tests/__init__.py", line 301, in assertEqual
          return super(NeoTestBase, self).assertEqual(first, second, msg=msg)
      failureException: '\x03\xd8\x85H\xbffp\xbb' != '\x03\xd8\x85H\xbfs\x0b\xdd'
      
      (cherry picked from commit f4cb59d2)
      a1418c9d
    • Julien Muchembled's avatar
      client: fix race with invalidations when starting a new transaction on ZODB 5 · 96a5c01f
      Julien Muchembled authored
      This requires ZODB >= 5.6.0
      
      (cherry picked from commit a7d101ec)
      96a5c01f
    • Julien Muchembled's avatar
      Code clean-up, comment fixes · fa7fbad6
      Julien Muchembled authored
      (cherry picked from commit 43029be2)
      fa7fbad6
  2. 22 May, 2020 2 commits
    • Julien Muchembled's avatar
      master: fix crash in STARTING_BACKUP when connecting to an upstream secondary master · 011eba12
      Julien Muchembled authored
      This fixes the following assertion:
      
        Traceback (most recent call last):
          File "neo/master/app.py", line 172, in run
            self._run()
          File "neo/master/app.py", line 182, in _run
            self.playPrimaryRole()
          File "neo/master/app.py", line 302, in playPrimaryRole
            self.backup_app.provideService())
          File "neo/master/backup_app.py", line 114, in provideService
            node, conn = bootstrap.getPrimaryConnection()
          File "neo/lib/bootstrap.py", line 74, in getPrimaryConnection
            poll(1)
          File "neo/lib/event.py", line 160, in poll
            to_process.process()
          File "neo/lib/connection.py", line 504, in process
            self._handlers.handle(self, self._queue.pop(0))
          File "neo/lib/connection.py", line 92, in handle
            self._handle(connection, packet)
          File "neo/lib/connection.py", line 107, in _handle
            pending[0][1].packetReceived(connection, packet)
          File "neo/lib/handler.py", line 125, in packetReceived
            self.dispatch(*args)
          File "neo/lib/handler.py", line 75, in dispatch
            method(conn, *args, **kw)
          File "neo/lib/handler.py", line 159, in notPrimaryMaster
            assert primary != self.app.server
        AttributeError: 'BackupApplication' object has no attribute 'server'
      
      (cherry picked from commit dba07e72)
      011eba12
    • Julien Muchembled's avatar
      mysql: workaround for MDEV-20693 · 70387981
      Julien Muchembled authored
      70387981
  3. 27 Apr, 2019 11 commits
    • Julien Muchembled's avatar
      master: reject drop/tweak ctl commands that could lead to unwanted status · 55a6dd0f
      Julien Muchembled authored
      The following 2 operations can be onerous and they should not be
      directly usable without some kind of confirmation by the user:
      - Dropping a node now requires to first stop it.
      - Tweaking does not exclude anymore automatically DOWN nodes,
        because a node could go DOWN between the moment the user sends
        the command to tweak and the actual tweak by the master.
      55a6dd0f
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      ab082d7e
    • Julien Muchembled's avatar
      tweak: add option to simulate · 2a27239d
      Julien Muchembled authored
      Initially, I wanted to do the simulation inside neoctl but it has no knowledge
      of the topology (the master don't send devpath values of storage nodes).
      Therefore, the work is delegated to the master node, which implies a change
      of the protocol.
      2a27239d
    • Julien Muchembled's avatar
      3839d224
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      Better error reporting from the master to neoctl for denied requests · c2c9e99d
      Julien Muchembled authored
      This stops abusing ProtocolError, which disconnects the admin node needlessly.
      
      The many 'if ... raise RuntimeError' in neo/neoctl/neoctl.py
      could be turned into assertions.
      c2c9e99d
    • Julien Muchembled's avatar
      21190ee7
    • Julien Muchembled's avatar
      Make the number of replicas modifiable when the cluster is running · ef5fc508
      Julien Muchembled authored
      neoctl gets a new command to change the number of replicas.
      
      The number of replicas becomes a new partition table attribute and
      like the PT id, it is stored in the config table. On the other side,
      the configuration value for the number of partitions is dropped,
      since it can be computed from the partition table, which is
      always stored in full.
      
      The -p/-r master options now only apply at database creation.
      
      Some implementation notes:
      
      - The protocol is slightly optimized in that the master now sends
        automatically the whole partition tables to the admin & client
        nodes upon connection, like for storage nodes.
        This makes the protocol more consistent, and the master is the
        only remaining node requesting partition tables, during recovery.
      
      - Some parts become tricky because app.pt can be None in more cases.
        For example, the extra condition in NodeManager.update
        (before app.pt.dropNode) was added for this is the reason.
        Or the 'loadPartitionTable' method (storage) that is not inlined
        because of unit tests.
        Overall, this commit simplifies more than it complicates.
      
      - In the master handlers, we stop hijacking the 'connectionCompleted'
        method for tasks to be performed (often send the full partition
        table) on handler switches.
      
      - The admin's 'bootstrapped' flag could have been removed earlier:
        race conditions can't happen since the AskNodeInformation packet
        was removed (commit d048a52d).
      ef5fc508
    • Julien Muchembled's avatar
      New --new-nid storage option for fast cloning · 27e3f620
      Julien Muchembled authored
      It is often faster to set up replicas by stopping a node (and any
      underlying database server like MariaDB) and do a raw copy of the
      database (e.g. with rsync). So far, it required to stop the whole
      cluster and use tools like 'mysql' or sqlite3' to edit:
      - the 'pt' table in databases,
      - the 'config.nid' values of the new nodes.
      
      With this new option, if you already have 1 replica, you can set up
      new replicas with such fast raw copy, and without interruption of
      service. Obviously, this implies less redundancy during the operation.
      27e3f620
    • Julien Muchembled's avatar
      qa: fix 2 tests with ZODB5 · 64e02391
      Julien Muchembled authored
      64e02391
  4. 26 Apr, 2019 4 commits
  5. 16 Apr, 2019 5 commits
  6. 05 Apr, 2019 3 commits
  7. 01 Apr, 2019 1 commit
  8. 21 Mar, 2019 2 commits
  9. 16 Mar, 2019 1 commit
    • Julien Muchembled's avatar
      importer: fix possible data loss on writeback · e387ad59
      Julien Muchembled authored
      If the source DB is lost during the import and then restored from a backup,
      all new transactions have to written back again on resume. It is the most
      common case for which the writeback hits the maximum number of transactions
      per partition to process at each iteration; the previous code was buggy in
      that it could skip transactions.
      e387ad59
  10. 13 Mar, 2019 6 commits
  11. 11 Mar, 2019 2 commits