1. 04 Jun, 2021 1 commit
    • Julien Muchembled's avatar
      admin: fix crash if not operational and a downstream cluster is RUNNING · 7f81ac2d
      Julien Muchembled authored
      Traceback (most recent call last):
        ...
        File ".../neo/lib/handler.py", line 75, in dispatch
          method(conn, *args, **kw)
        File ".../neo/admin/handler.py", line 174, in wrapper
          return func(self, name, *args, **kw)
        File ".../neo/admin/handler.py", line 190, in notifyMonitorInformation
          self.app.updateMonitorInformation(name, **info)
        File ".../neo/admin/app.py", line 290, in updateMonitorInformation
          self._notify(self.operational)
        File ".../neo/admin/app.py", line 315, in _notify
          body += '', name, '    ' + backup.formatSummary(upstream)[1]
        File ".../neo/admin/app.py", line 83, in formatSummary
          tid = self.ltid
      AttributeError: 'Backup' object has no attribute 'ltid'
      7f81ac2d
  2. 02 Apr, 2021 4 commits
  3. 22 Mar, 2021 1 commit
  4. 04 Mar, 2021 1 commit
  5. 15 Jan, 2021 1 commit
    • Kirill Smelkov's avatar
      ssl: Don't ignore non-ragged EOF · 061cd5d8
      Kirill Smelkov authored
      Testing NEO/go client wrt NEO/py server revealed a bug in NEO/py SSL
      handling: proper non-ragged EOF from a peer is ignored, and so leads to
      hang in infinite loop inside _SSL.receive with read_buf memory growing
      indefinitely. Details are below:
      
      NEO/py wraps raw sockets with
      
      	ssl.wrap_socket(suppress_ragged_eofs=False)
      
      which instructs SSL layer to convert unexpected EOF when receiving a TLS
      record into SSLEOFError exception. However when remote peer properly
      closes its side of the connection, socket.read() still returns b'' to
      report non-ragged regular EOF:
      
      https://github.com/python/cpython/blob/v2.7.18/Lib/ssl.py#L630-L650
      
      The code was handling SSLEOFError but not b'' return from socket recv.
      Thus after NEO/go client was disconnecting and properly closing its side
      of the connection, the code started to loop indefinitely in _SSL.receive
      under `while 1` with  b'' returned by self.socket.recv() appended to
      read_buf again and again.
      
      -> Fix it by detecting non-ragged EOF as well and, similarly to how
      SSLEOFError is handled, converting them into self._error('recv', None).
      
      See merge request !17
      061cd5d8
  6. 11 Jan, 2021 2 commits
  7. 02 Oct, 2020 1 commit
    • Julien Muchembled's avatar
      Fix handling of -m/--masters arg · fa63d856
      Julien Muchembled authored
      For the master, the purpose of -m/--masters is to specify addresses
      of other master nodes, since its own address is already known via
      -b/--bind. Therefore, an empty value for -m/--masters is valid.
      The user remains free to repeat the -b value in -m.
      
      More generally, a node may choose to only specify master addresses
      via -D/--dynamic-master-list, so the check that at least one master
      address is specified is moved where the NodeManager is expected to be
      initialized.
      fa63d856
  8. 25 Sep, 2020 3 commits
    • Julien Muchembled's avatar
      New algorithm for deadlock avoidance · 5e7f34d2
      Julien Muchembled authored
      The time complexity of previous one was too bad. With several tens of
      concurrent transactions, we saw commits take minutes to complete and
      the whole application looked frozen.
      
      This new algorithm is much simpler. Instead of asking the oldest
      transaction to somewhat restart (we used the "rebase" term because
      the concept was similar to what git-rebase does), the storage gives
      it priority and the newest is asked to relock (this request is ignored
      if vote already happened, which means there was actually no deadlock).
      
      testLocklessWriteDuringConflictResolution was initially more complex
      because Transaction.written (client) ignored KeyError (which is not the
      case anymore since commit 8ef1ddba).
      5e7f34d2
    • Julien Muchembled's avatar
      qa: deindent code · d98b576c
      Julien Muchembled authored
      d98b576c
    • Julien Muchembled's avatar
      Update comments · dbf128b7
      Julien Muchembled authored
      dbf128b7
  9. 21 Aug, 2020 1 commit
  10. 24 Jun, 2020 1 commit
  11. 12 Jun, 2020 1 commit
    • Julien Muchembled's avatar
      qa: skip broken ZODB test · f4cb59d2
      Julien Muchembled authored
      ======================================================================
      FAIL: check_tid_ordering_w_commit (neo.tests.zodb.testBasic.BasicTests)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "ZODB/tests/BasicStorage.py", line 397, in check_tid_ordering_w_commit
          self.assertEqual(results.pop('lastTransaction'), tids[1])
        File "neo/tests/__init__.py", line 301, in assertEqual
          return super(NeoTestBase, self).assertEqual(first, second, msg=msg)
      failureException: '\x03\xd8\x85H\xbffp\xbb' != '\x03\xd8\x85H\xbfs\x0b\xdd'
      f4cb59d2
  12. 11 Jun, 2020 1 commit
  13. 16 Mar, 2020 1 commit
  14. 10 Jan, 2020 1 commit
    • Julien Muchembled's avatar
      master: fix crash of backup master when disconnected from upstream while serving clients · 7e8ca9ec
      Julien Muchembled authored
      This fixes:
      
        Traceback (most recent call last):
          File "neo/master/app.py", line 172, in run
            self._run()
          File "neo/master/app.py", line 182, in _run
            self.playPrimaryRole()
          File "neo/master/app.py", line 314, in playPrimaryRole
            self.backup_app.provideService())
          File "neo/master/backup_app.py", line 101, in provideService
            app.changeClusterState(ClusterStates.STARTING_BACKUP)
          File "neo/master/app.py", line 474, in changeClusterState
            ) or not node.isClient(), (state, node)
        AssertionError: (<EnumItem STARTING_BACKUP (4)>, <ClientNode(uuid=C1, state=RUNNING, connection=<ServerConnection(nid=C1, address=127.0.0.1:52430, handler=ClientReadOnlyServiceHandler, fd=59, on_close=onConnectionClosed, server) at 7f38f5628390>) at 7f38f5628ad0>)
      7e8ca9ec
  15. 22 Oct, 2019 1 commit
  16. 17 Oct, 2019 1 commit
  17. 14 Oct, 2019 3 commits
  18. 16 Aug, 2019 1 commit
    • Julien Muchembled's avatar
      New feature: monitoring · e434c253
      Julien Muchembled authored
      This task is done by the admin node, in 2 possible ways:
      - email notifications, as soon as some state change;
      - new 'neoctl print summary' command that can be used periodically
        to check the health of the database.
      They report the same information.
      
      About backup clusters:
      
      The admin of the main cluster also monitors selected backup clusters,
      with the help of their admin nodes.
      
      Internally, when a backup master node connects to the upstream master node,
      it receives the address of the upstream admin node and forwards it to its
      admin node, which is therefore able to connect to the upstream admin node.
      So the 2 admin nodes remain connected and communicate in 2 ways:
      - the backup node notifies upstream about the health of the backup cluster;
      - the upstream node queries the backup node periodically to check whether
        replication is not too late.
      
      TODO:
      
      A few things are hard-coded and we may want to configure them:
      - backup lateness is checked every 10 min;
      - backup is expected to never be late.
      
      There's also no delay to prevent 2 consecutive emails from having the same
      Date: (unfortunately, the RFC 5322 does not allow sub-second precision),
      in which case the MUA can display them in random order. This is mostly
      confusing when one notification is OK and the other is not, because one
      may wonder if there's a new problem.
      e434c253
  19. 05 Jun, 2019 1 commit
    • Julien Muchembled's avatar
      Introduce extra node properties · 82c142c4
      Julien Muchembled authored
      Explicit fields in RequestIdentification are only suitable for the actual
      identification or for properties that most nodes have.
      
      But some current (and future) features require to pass values (always and
      as soon as possible) for tasks that are unrelated to identification.
      82c142c4
  20. 24 May, 2019 1 commit
  21. 20 May, 2019 1 commit
  22. 09 May, 2019 2 commits
  23. 30 Apr, 2019 2 commits
    • Julien Muchembled's avatar
      importer: fix writeback of transactions during which readCurrent() was used · 042f5ac0
      Julien Muchembled authored
      Contrary to FileStorage, NEO remembers uses of readCurrent().
      042f5ac0
    • Julien Muchembled's avatar
      master: fix crash in STARTING_BACKUP when connecting to an upstream secondary master · dba07e72
      Julien Muchembled authored
      This fixes the following assertion:
      
        Traceback (most recent call last):
          File "neo/master/app.py", line 172, in run
            self._run()
          File "neo/master/app.py", line 182, in _run
            self.playPrimaryRole()
          File "neo/master/app.py", line 302, in playPrimaryRole
            self.backup_app.provideService())
          File "neo/master/backup_app.py", line 114, in provideService
            node, conn = bootstrap.getPrimaryConnection()
          File "neo/lib/bootstrap.py", line 74, in getPrimaryConnection
            poll(1)
          File "neo/lib/event.py", line 160, in poll
            to_process.process()
          File "neo/lib/connection.py", line 504, in process
            self._handlers.handle(self, self._queue.pop(0))
          File "neo/lib/connection.py", line 92, in handle
            self._handle(connection, packet)
          File "neo/lib/connection.py", line 107, in _handle
            pending[0][1].packetReceived(connection, packet)
          File "neo/lib/handler.py", line 125, in packetReceived
            self.dispatch(*args)
          File "neo/lib/handler.py", line 75, in dispatch
            method(conn, *args, **kw)
          File "neo/lib/handler.py", line 159, in notPrimaryMaster
            assert primary != self.app.server
        AttributeError: 'BackupApplication' object has no attribute 'server'
      dba07e72
  24. 28 Apr, 2019 2 commits
    • Julien Muchembled's avatar
      qa: add testrunner options to dump/check the format of network packets · e3cd5c5b
      Julien Muchembled authored
      With the switch to msgpack, there was no schema anymore whereas it was
      sometimes used for both automatic conversion (e.g. the last argument of
      AskStoreTransaction must now be explicitly cast to list) and type checking.
      
      This somewhat reintroduces a kind of schema that:
      - is used by the test suite for type checking
      - can be generated automatically from the test suite
        when one change the procotol
      e3cd5c5b
    • Julien Muchembled's avatar
      protocol: switch to msgpack for packet serialization · 9d0bf97a
      Julien Muchembled authored
      Not only for performance reasons (at least 3% faster) but also because of
      several ugly things in the way packets were defined:
      - packet field names, which are only documentary; for roots fields,
        they even just duplicate the packet names
      - a lot of repetitions for packet names, and even confusion between the name
        of the packet definition and the name of the actual notify/request packet
      - the need to implement field types for anything, like PByte to support new
        compression formats, since PBoolean is not enough
      
      neo/lib/protocol.py is now much smaller.
      9d0bf97a
  25. 27 Apr, 2019 5 commits